VoLUME 64 WHOLE No. 314 
NuMBER 8 1950 


Psychological Monographs: 
General and Applied 


Combining the Applied Psychology Monographs and the Archives of Psychology 
with the Psychological Monographs 


HERBERT S. CONRAD, Editor 


The Validity of a Multiple-Choice 
Projective Test in Psychopathological 
Screening, 


By 
MARTIN SINGER 


Veterans Administration Hospital 
Northport, Long Island, N.Y. 


A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of 
Doctor of Philosophy in the University of Pennsylvania 


Accepted for publication, November 7, 1949 


Price $1.00 


Published by 
‘THE AMERICAN PSYCHOLOGICAL ASSOCIATION, INC. 
1515 MASSACHUSETTS AVE., N.W., WASHINGTON 5, D.C. 


in 
niv. 

1, 
ing 
rk, ‘3 
On 
10, 
ery 

‘ 
ses 
28, 

icy. 
ner. 

gen 
by 
of 

48. 
nce. 
No. 
eim 
934, 
str., 


COPYRIGHT, 1950, BY THE 


AMERICAN PSYCHOLOGICAL ASSOCIATION 


L 
| 
( 


ACKNOWLEDGEMENTS 


HE WRITER wishes to express his deep gratitude to Professor Malcolm G. Preston 

for his invaluable aid in the formulation and execution of this study. Ac- 
knowledgements must also be made to Professor Robert A. Brotemarkle, chairman, 
and to the other members of the department of psychology, University of Pennsyl- 
vania, and to Drs. Richard H. Paynter, Morton L. Wadsworth and Earl E. Swartz- 
lander of the Veterans Administration for their interest and help. 

The necessary subjects for this study could not have been obtained without the 
cooperation of very many individuals from various institutions. Space limitations 
permit only the following token acknowledgements: 

Brooklyn State Hospital: Drs. Clarence Bellinger, Nathan Beckenstein, Louis 
Mantone, James Rappa, Paul Tarantalla, Sidney Tamarin, and Joseph Villara. 

Jewish Hospital of Brooklyn (Rehabilitation Clinic): Dr. Joseph L. Abrahamson 
and Mrs, Eugenia Gorman. 


Mt. Sinai Hospital (Veterans Clinic): Dr. Paul Goolker, Mrs. Roslyn Braun, and 
Miss Ruth Cuker. 


Post Graduate Hospital (Veterans Clinic): Dr. William D. Sherwood and Miss 
Norrie O’Brien. 

Veterans Administration: Dr. Helen M. Campbell, Dr. Irwin J. Cohen, Mr. John 
W. Green, Mr. Robert Lackey, Dr. Bernard Locke, Dr. Harry M. Harter, Dr. James 
Q. Holsopple, Mr. Thomas MacNamara, Dr. Charles M. Morris, Mr. Martin Munzer, 
and Dr, Morris J. Tissenbaum. 

Finally, thanks are due to Miss Margaret J. Crookhorn and the personnel in the 
Mental Hygiene Service of the New York Regional Office, Veterans Administration, 
for their kind assistance. 


MARTIN SINGER 


CONTENTS 


I. INTRODUCTION 


. EXPERIMENTAL DESIGN 


. CONCLUSIONS 


. APPENDIX 


III. DESCRIPTION OF THE POPULATION TESTED : 

¥ : 


THE VALIDITY OF A MULTIPLE-CHOICE PROJECTIVE TEST 
IN PSYCHOPATHOLOGICAL SCREENING! 


HERE is marked need for a test which 
Tea be rapidly administered and 
scored, and which has sufficient validity 
for use as a screening device in the de- 
tection of psychopathology. The present 
study is an attempt to determine whether 
the projective test approach? may be util- 
ized for this purpose. 


In the past decade, more and more clinical 
psychologists have turned to the use of projec- 
tive tests in their attempts to identify psycho- 
pathology. Most of the projective tests have been 
standardized under conditions of individual ad- 
ministration and free association responses. Such 
tests have not been suitable as screening instru- 
ments since a great deal of time is required 
by a highly skilled psychologist to administer 
and interpret the results for each subject. How- 
ever, there have been some projective tests in 
which the subjects selected their responses from 
several multiple choices listed, permitting rapid 
and objective scoring by untrained personnel.’ 
The most widely used test of this type has been 
the Harrower Multiple Choice Rorschach (19). 


‘Sponsored by the Veterans Administration 
and published with the approval of the Chief 
Medical Director. The statements and conclu- 
sions published by the author are a result of his 
own study and do not necessarily reflect the 
opinion or policy of the Veterans Administra- 
uion, 

* Comprehensive discussions of the projective 
methods may be found in Frank (12), Sargent 
(46) and White (57) . Each of a series of reviews 
by Symonds and Samuel (49), Symonds and 
Krugman (48), and Hertz, Ellis and Symonds 
(21) has summarized the literature published 
since the preceding review. Briefly, the projec- 
tive test approach is defined as one in which 
the subject is required to structure an ambigu- 
ous stimulus. In doing so, he is presumed to 
reveal his own personality structure. 

*There have also been some attempts to 
quantify tests calling for free association re- 
sponses, by organizing the responses into various 
categories which are scored linearly; e.g., the 
Rorschach Inspection Technique as originated by 
Munroe (37) and modified by Ross and Ross 
(45). It is felt that the validity of such tests as 
screening devices should be investigated if a 
completely objective. scoring system has been 
proven inadequate. 


I. INTRODUCTION 


Inkblots were also employed as stimuli by Ter- 
man (52) and by Harrower and Grinker (4, 17). 
Pictures were used as stimuli by Clark (5), 
Franck (11), amd Horowicz (23), Murray and 
Stein (38), Szondi (7, 41) and Temple and 
Amen (8, 51). Single words served as- stimuli 
for Crown (6), Malamud (31), Maller (34', 
Mittelman and Brodman (36) and Pressey (39, 
40), while story plots were used by Kelly and 
Bishop (27), Taylor (50) and Wolfenstein 
59) 

A review of the literature indicates that most 
of the multiple choice projective tests achieved 
little discrimination. In those cases where a fair 
amount of validity was reported, faults in the 
experimental design, such as improper cross 
validation, use of extreme groups, or testing 
of too few subjects, made the validity coeffi- 
cients suspect. On the other hand, none of the 
studies which found no discrimination was suffi- 
ciently comprehensive to warrant a negative con- 
clusion; namely, that the limitations inherent in 
any attempt to restrict responses to those listed 
on a test blank preclude valid discrimination 
with a projective test. Therefore, the present 
experiment was undertaken in an effort to 
determine the potentialities of the multiple 
choice projective test approach in the detection 
of psychopathology. 


A multiple choice projective test was 
constructed and administered under 
what were considered “optimum” con- 
ditions; i.e., under those conditions 
which seem most likely to result in the 
detection of psychopathology. If, with 
such a test, a fair degree of validity were 
obtained, then further experimentation 
with this approach would seem _ indi- 
cated. On the other hand, if very little 
or no validity were obtained, then it 
would appear more profitable to explore 
other approaches—ualess, of course, it 
was established that optimum conditions 
did not exist so that factors which might 
lead to radically different results were 
not included in the present experimental 
design. 


Wo AREAS where optimum condi- 
must exist may be distin- 
guished: first, in the selection of the par- 
ticular psychopathology to be identified, 
and second, in the test methods used to 
identify this psychopathology. 


A. Type or PsYCHOPATHOLOGY SELECTED 
FOR IDENTIFICATION 


In selecting the psychopathology to 
be identified, it is recognized that the 
grosser such psychopathology is, the 
more optimum conditions will be for 
valid identification. However, the gross- 
ness of the psychopathology must be 
limited so that useful conclusions may be 
drawn from the results. For example, the 
successful identification of deteriorated 
psychotics by a multiple choice projec- 
tive test would be of little value. A 
screening device is not needed for this 
purpose, and no inferences could be 
drawn as to the probabilities of finer and 
more useful psychopathological identi- 
fication. Therefore, the grossness of the 
psychopathology selected for identifica- 
tion in the present investigation was 
limited to such distinctions as would be 
required of a screening device. 

Since projective tests have been used 
most successfully in the detection of 
psychopathological diagnoses, the at- 
tempt was made in the present investiga- 
tion to differentiate among three diag- 
nostic categories: normals, anxiety reac- 
tions, and schizophrenics. These particu- 
lar categories were selected because they 
provided adequate distinctions and be- 
cause their frequency made available a 
sufficiently large number of subjects in 
each category.! A more detailed consid- 


*It was recognized that specific situations in 
which a test would be required to discriminate 
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eration of these diagnostic categories will 
be presented in section III. 


B. THe DEVELOPMENT OF A MULTIPLE- 
Cuoice Projective TEst 


The review of previous research with 
multiple choice projective tests disclosed 
no test with sufficiently promising results 
to warrant further exploration of that 
particular instrument. Therefore, a new 
test was constructed, based on the results 
obtained with individual, group, and 
multiple choice projective tests, and up- 
on recommendations of various authori- 
ties in the field of projective testing. In 
this test, certain conditions were fixed 
since time considerations limited the 
length of the test. However, it was pos- 
sible to vary other factors within the test 
so that the effect of such variation could 
be estimated. 


1. Factors Not Varied 


a. Stimuli: The Rorschach inkblots 
were selected as the stimuli because the 
Rorschach appears to be the best projec- 
tive test for the prediction of psycho- 
pathology. 

According to Rapaport and Schafer, “The use 


of the Rorschach Test, as one in a battery of 
tests, in a clinical-psychiatric setting for pur- 


only among subjects in these three diagnostic 
classifications would rarely arise. However, the 
purpose of the experiment was not to develop 
a test for practical use, but to determine whether 
the multiple choice projective test approach is 
capable of valid psychopathological differentia- 
tion. If potential validity is established, then 
efforts can be made to develop a test for practical 
purposes; e.g., to screen inductees in the armed 
forces, to select employees in industry, or to dis- 
cover students with marked personality prob- 
lems. In order to limit the psychopathological 
deviation of subjects in this study to that which 
would exist in populations such as those men- 
tioned above, the most seriously disturbed anx- 
iety reactions and schizophrenics were not 
tested. (See section ITI.) 


poses of diagnoses and appraisal of adjustment 
shows... . that it is beyond doubt the most 
potent single diagnostic instrument clinical psy- 
chology possesses” (42, p. 73). Brody, in review- 
ing the literature on mental testing from 1938 
to 1942, declared: “Increased use of mental tests 
for the study of psychoses and allied disorders 
is conspicuous in recent psychiatric practice. 
This increase is most prominent first in the use 
of the Rorschach ink blots” (3, p. 127), Symonds 
and Krugman point out the superiority of the 
Rorschach indirectly with their statement: “Next 
to the Rorschach, the Thematic Apperception 
Test has claimed most attention. Indeed, in 
numerous testing programs the Thematic Ap- 
perception Test and the Rorschach are listed as 
the two preferred methods for the study of 
personality. Actually, much yet remains to be 
done in exploration of the use of the Thematic 
Apperception method before it can receive gen- 
eral clinical use” (48, p. 88). Michael and Buh- 
ler (35) found the Rorschach to be superior as 
a diagnostic instrument to a variety of other 
tests both projective and non-projective. 

While in confirmatory experiments (2, 15, 
26, 33, 47, 58) poor results were obtained with 
the Harrower Multiple Choice Rorschach, which 
used the Rorschach inkblots as stimuli, such re- 
sults might well have been caused by inade- 
quacies of the specific test rather than by the 
nature of the stimuli. Hunt and Stevenson have 
remarked that “. . . . had Harrower-Erickson 
followed the usual painstaking steps of item 
analysis and test validation, relying more on 
empirical investigation and less on a priori 
judgment in selecting her alternative responses 
and constructing her scoring key, the results 
might have been different” (24, p. 110). 


b. Method of administration: As the 
purpose of the present study was to con- 
struct and evaluate a test which could 
be rapidly administered, it was thought 
desirable to follow a group administra- 
tion procedure. The Rorschach slides 
were used since various investigators (18, 
20, 22, 29, 30) have found that responses 
made to these slides are comparable to 
those made to the individual cards. 
These slides were projected on a screen 
in a semi-darkened room. An alternative 
possibility, the use of booklets in which 
the Rorschach cards were printed, was 
eliminated because of the expense of 
printing such booklets, and because the 
time of stimulus presentation would 
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have been difficult to control, particu- 
larly with the non-normal subjects. 

c. The use of multiple choice ques- 
tions: It is only with some form of a 
multiple choice question that the re- 
quirement of rapid objective scoring can 
be met. 

d. The empirical determination of the 
scoring weights to be given each re- 
sponse:* Results obtained with the Har- 
rower Multiple Choice Rorschach by 
Due, Wright and Wright (9), Malamud 
and Malamud (31, 32), Lawshe and 
Forster (28), and Blair and Hoehn (2) 
have clearly indicated that a reponse to 
a multiple choice Rorschach might not 
have the same psychopathological sig- 
nificance as the same response to the free 
association Rorschach. Therefore, the 
psychopathological significance of multi- 
ple choice Rorschach responses must be 
determined independently. 


2. Factors Varied within the Test 


The factors which were varied are 
summarized below. They will be dis- 
cussed more fully in subsequent sections. 


a. Structure of the questions: 

(1). A “relative” versus an “absolute” deci- 
sion by the subject. In some questions the sub- 
ject was required to select one of four percepts 
(relative decision), while in other questions the 
subject was required to rate a single percept on a 
five point rating scale (absolute decision) . 

(2). Differences in the types of alternatives 


* At this point, it would seem advisable to de- 
fine several terms which ordinarily may have 
different connotations. 

A percept is a possible description of the 
whole slide or some part of the slide. It may be 
specific, such as “A bat”; abstract, such as 
“Death”; or a denial, such as “Nothing at all.” 

An alternative is one of the multiple choices 
listed in a question. It may be a percept, as when 
the subject is required to select one of four per- 
cepts listed, or it may be a rating, as when the 
subject is required to select one of five ratings 
on a rating scale. 

A response is the answer a subject makes to a 
question. A subject's response can be either one 
of the alternatives listed in a question, or it may 
be a failure to select any alternative. 
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included in @ qiiestion. The types of alternatives 
were varied to form the following question “de- 
signs.” 

(a). Design 1 (forced “normal” choice) 
contained what were considered on a_ priori 
grounds to be two percepts characteristic of 
normals, one percept characteristic of anxiety 
reactions, and one percept characteristic of 
schizophrenics.’ 

(b). Design II (unlimited choice) con- 
tained one “normal” percept, one “anxiety reac- 
tion” percept, one “schizophrenic” percept, and 
one percept worded “Something else.” 

(c). Design III (forced “abnormal” 
choice) contained two “anxiety reaction’ per- 
cepts and two “schizophrenic” percepts. 

b. Time of stimulus presentation: As subjects’ 
immediate perceptions might differ from their 
more reflective perceptions, the time of stimulus 
presentation included both a brief exposure of 
the slides and an exposure which gave the sub- 
jects as much time as they required. 

c. Length of test: The total test battery was 
made long enough so that information might be 
obtained as to the effect of increasing the size 
of the test by comparing the results obtained 
from sections of the test with those obtained 
from the total test. 

d. Method of determining scoring weights for 
each response: While all methods depended upon 


*The diagnostic significance of the various 
ea had to be determined on an a priori 

is because no empirically established norms 
were available. Furthermore, it was necessary 
to base these a priori judgments on the results 
obtained with the free association Rorschach, 
since so little was known about multiple choice 
Rorschach responses. 

The writer would like to express his apprecia- 
tion to Drs. Bruno Klopfer and Zygmunt A. Pio- 
trowski for their assistance in the selection of 
the percepts believed to be characteristic of the 
different a categories. The method by 
which the different percepts were chosen as 
characteristic of normals, anxiety reactions, and 
schizophrenics for the different inkblots is illus- 
trated below by the way in which the “normal” 
percepts were selected for the first Rorschach 
blot. 

The writer, after consultation with various 
experienced Rorschach examiners, drew up a list 
of about 15 percepts which were considered to 
be the most characteristic of normals. Drs. 
Klopfer and Piotrowski examined this list inde- 
pendently and each checked off the eight per- 
cepts they considered most characteristic. In 
some cases, they substituted new percepts as 
better choices than those listed. The writer used 
the selection of Drs. Klopfer and Piotrowski as 
a basis for choosing nine percepts for each card. 
Then, when the structure of a question called 
for a “normal” cept, one of these nine per- 
cepts was used. Which particular one was selected 
was determined in a relatively random fashion. 


SINGER 


the relative percentage of responses made by 
two groups, various formulae were employed 
Which required varying amounts of difference 
between the two percentages for responses to be 
given scoring weights. 


3. The Final Test Battery 


Having determined the _psychopa- 
thology to be identified (normal, anxiety 
reaction, and schizophrenia as discussed 
in section II, A) and the general composi- 
tion of the test to be used for this identi- 
fication, the next step was to construct 
a specific test. After experimentation 
with preliminary groups, the following 
test battery, termed the Singer Multiple 
Choice Rorschach (SMCR) to differenti- 
ate it from the Harrower Multiple 
Choice Rorschach, was developed. (A 
copy of the SMCR is shown in Appendix 
A.) The battery consisted of three tests. 


a. Test A: 

(1). Number of questions: 30 

(2). Number of percepts listed in a ques- 
tion: 4 

(3). Type of percepts in a question: There 
were 10 questions each of Designs I, II and III. 
(The three questions for each slide included one 
of each design. The order in which the designs 
were presented was varied systematically.) 

(4). Response by the subject: The subject 
was required to select one of the four percepts 
in each question. 

(5). Time of stimulus presentation: A slide 
was shown for 10 seconds for each question. Sub- 
jects were then allowed a few seconds more in 
which to record their response if they had not 
already done so. 

b. Test B: This test was exactly the same as 
Test A with the following exceptions: 

(1). Different percepts were used: However, 
the diagnostic significance of each percept paral- 
leled that in Test A. 

(2). Time of stimulus presentations: The 
subjects were allowed one and one-half minutes 
to answer the three questions for each slide. 
Preliminary experimentation had shown that 
when given an unlimited amount of time, over 
95% of the subjects finished within the above 
time limit. 

c. Test C: 

(1). Number of questions: 100 

(2). Number of percepts listed in a ques- 
tion: 1 

(3). Types of percepts in a question: There 


vever, 


paral- 


were 10 questions for each slide, divided as fol- 
lows: 

(a). Four “normal” percepts. 

(b). Three “anxiety reaction” percepts. 

(c). Three “schizophrenic” percepts, one 
of which was “Nothing at all.” 

(4). Response by the subject: Each ques- 
tion (percept) was rated by the subject on a 
five point absolute rating scale as follows: 

(a). Exactly like it; 

(b). Very much like it; 
(c). Somewhat like it; 
(d). Only a little like it; 
(e). Not at all like it. 

The term “like it” referred to how much 
either the whole slide or any part of it re- 
sembled the percepts listed. 

(5). Time of stimulus presentation: 

(a). Slide I. Three minutes. 
(b). Slide II. Two and one-half minutes. 
(c). Slides III through X. Two minutes. 

The decreasing time limits were established 
because it was found that the subjects required 
less time to answer a question as they became 
familiar with the rating technique. Preliminary 
experimentation indicated that over 95% of the 
subjects finished within the above time limits 
when they were allowed unlimited time. 

For Slide I subjects were warned when they 
had one minute left, while for the other slides 
they were informed when there were thirty sec- 
onds remaining. 


The differences among the various 
tests may be observed more clearly in 
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Table 1. The effect of these differences 
will be discussed in section IV. 


4. Administration of the Test 


The regular 314” by 414” Rorschach 
slides, purchased from the Psychological 
Corporation, were projected on a screen 
by a lantern projector illuminated by a 
lamp of at least 500 watts. 


The screen image was generally at least 4’ by 
5’ in size. In a few cases the image was slightly 
smaller because the size of the testing room 
made it impossible to focus a larger clear image. 
However, at such times the subjects were seated 
slightly nearer to the screen than in the cases 
where the 4’ by 5’ image was employed. It was 
possible, by having the light enter from the back 
of the room, to keep the image on the screen 
quite clear, while enough light existed for the 
subjects to read the alternatives on their test 
blanks. The room was kept in this semi-dark con- 
dition throughout the test. Subjects were seated 
at least 12 feet from the screen and at an angle 
of less than go° from the center of the screen. 


Before each test began, the subjects 
were given the answer sheets which listed 
the various possible responses for that 
test. They were told that slides would be 


TABLE 1 


' CHARACTERISTICS OF THE DIFFERENT TESTS IN THE SMCR 


Differences in Types of Percepts* 


Time of . Design I Design II Design III 
Test Stimulus Total 

Exposure J No. | Percepts | No. | Percepts | No. | Percepts 

of ina of ina |, of ina 
Quest.| Quest. | Quest.) Quest. | Quest.) Quest. 
A | Brief Select 1 of 4 per- 10 N-2 10 N-1 10 AR-2 30 
cepts AR-1 AR-1 S-2 
S-1 S-1 


Relatively | Select 1 of 4 per- | 10 
Unlimited | cepts 


AR-1 | 
S-1 | S-1 


N-2 10 N-1 10 AR-2 30 
AR-1 S-2 


Relatively 


Rate each percept | 40 
Unlimited 


on an absolute rat- 
ing scale. 


N-1 30 AR-1 30 S-1 100 


* N stands for a percept considered on an a priori basis to be characteristic of normals, AR for one 


characteristic of anxiety reactions, and S for one characteristic of schizophrenics. The ‘‘?” stands for 


the alternative “Something else.” 
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projected on the screen, and that their 
task was to select a response while the 
slide was on the screen. No references 
were made to the Rorschach or to ink- 
blots. The specific procedure for each 
test was as follows: 


a. Test A. The first Rorschach slide was pro- 
jected on the screen for ten seconds, during 
which time the subjects checked one of the 
four alternatives in Question 1. The slide was 
taken off the screen and after a few seconds 
the same slide was again projected for 10 seconds 
while the subjects answered Question 2. After 
an interval of a few seconds, the slide was shown 
for another 10 seconds while the subject an- 
swered Question 3. The second Rorschach slide 
was then shown for 10 seconds with the sub- 
jects responding to Question 4. This procedure 


continued, with a new slide being used for each 
three questions, until all go questions were 
answered. 

b. Test B. The first Rorschach slide was again 
projected on the screen while the subjects an- 
swered Questions 1A, 2A and 3A. The slide 
remained on the screen for go seconds while 
all three questions were answered. Then the 
second Rorschach slide was shown for go seconds 
while the subjects responded to the next three 
questions, and so on until all go questions were 
answered. 

c. Test C. The first Rorschach slide was shown 
on the screen for three minutes while the sub- 
jects rated all 10 percepts for that slide. The 
second slide was projected for two and a half 
minutes while the subjects rated the second 
10 percepts. Then the other slides were shown 
in order for two minutes each while the subjects 
rated the 10 percepts listed for each slide. 
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N THE original experimental design, it 

was planned to establish the scoring 
weight for each response empirically 
from the results of 150 subjects, consist- 
ing of 50 normals, 50 anxiety reactions, 
and 50 schizophrenics, and to cross vali- 
date those weights on a second set of 
subjects. (The term “set” will always 
refer to a group consisting of 50 subjects 
in each of the three diagnostic cate- 
gories.) However, since it proved impos- 
sible to obtain subjects in the various 
diagnostic categories who were thor- 
oughly evaluated, and since the scoring 
weights established from only 50 subjects 
in each category proved highly unstable, 
a third set of subjects was tested to pro- 
vide a larger sample upon which to 
establish scoring weights. After the test- 
ing had been completed, it was found 
possible to add a fourth set consisting of 
an “excess” of properly qualified subjects 
who were not needed for the third set, 
and subjects who had been excluded 
from previous sets because they did not 
quite meet all the requirements. 

Over 2,000 subjects were tested, as it 
frequently was necessary, for administra- 
tive reasons, to include extraneous sub- 
jects in a testing group. The total num- 
ber of groups tested was go, of which 39 
included only normals, 27 included only 
hospitalized schizophrenics, and the re- 
maining 24 contained both anxiety re- 
actions and out-patient schizophrenics. 
The results of 653 subjects, 250 normals, 
203 anxiety reactions, and 200 schizo- 
phrenics, were used. 


A. AGE AND EDUCATION 


Information was obtained as to the age and 
education of the subjects, and the three diag- 
nostic groups proved to be fairly homogeneous 
in these respects. The mean age of the normals 
was 25.0 years, that of the anxiety reactions was 
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30.0 years, and of the schizophrenics 28.4 years. 
Standard Deviations were 5.0, 6.6 and 6.4 re- 
spectively. There was a great deal of overlap 
among the groups with 93% of the normals, 87°%, 
of the anxiety reactions, and 87% of the schizo- 
phrenics falling between the ages of 20 and 35. 

The educational level of the subjects was de- 
termined by the highest grade completed in 
school. Information was lacking for 5 normals, 
6 anxiety reactions, and 21 schizophrenics, but 
there was no reason to believe that the educa- 
tional level of these subjects was much different 
from that of the others. The mean highest 
grades reached were 13.2 for the normals, 11.4 
for the anxiety reactions and 11.5 for the schizo- 
phrenics, with respective Standard Deviations 
of 2.0, 2.5 and 2.6, Seventy-eight per cent of 
the normals, 45% of the anxiety reactions, and 
46% of the schizophrenics were graduated from 
high school. The percentages for those complet- 
ing the second year of high school were 96%, 
71% and 72% respectively. The few subjects who 
had less than an eighth grade education (6%, 
of the anxiety reactions and 7% of the schizo- 
phrenics) were generally older or foreign born 


individuals. 

No attempt was made to equate edu- 
cational level, since the lower scholastic 
accomplishment is characteristic of the 
general anxiety reaction and _ schizo- 
phrenic population. It was felt that there 
was sufficient overlap among the three 
diagnostic groups to preclude any sub- 
stantial differences arising merely as a re- 
flection of variations in education, The 
possible effect of differences in age and 
amount of schooling has been evaluated 
in section V,B,8. 


B. DEscRIPTION OF SUBJECTS IN EACH 
D1aGNostTic CATEGORY 
The general characteristics of the sub- 
jects in each diagnostic category will be 
discussed first; then the composition of 
the four different sets will be presented. 


1. Normals 


Subjects in this group were drawn 
from veterans who came to the New 
York Regional Office of the Veterans Ad- 
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ministration for a wide variety of non- 
psychiatric reasons (e.g., to obtain in- 
formation relating to insurance, benefits 
under the G.I. Bill of Rights, medical 
or dental treatment, etc.). All such vet- 
erans were first required to visit the Con- 
tact Division where they were inter- 
viewed and then routed to the appropri- 
ate section of the VA. Since the flow of 
veterans varied greatly during the week, 
at peak periods it was necessary for them 
to wait an hour or more before being 
interviewed. At this time, an announce- 
ment was made over a loud speaker ask- 
ing male veterans of World War II to 
volunteer to take a personality test while 
they were waiting. All such veterans who 
took the test were identified merely by a 
number which they placed on each page 
of their test blank. This was done to in- 
sure honesty of response as much as pos- 
sible. 

Subjects were selected as normals on 
the basis of their responses to two ques- 
tionnaires, one of the Cornell Selectee 
Index, Form N (53, 54, 55, 56), and the 
other a six-item questionnaire (listed in 
Appendix B) which was _ validated’ 
against the Cornell Index, in an effort 
to include the best adjusted individuals 
in this group. 


2. Anxiety Reactions 


Subjects in this category were obtained 
from the VA Mental Hygiene Services of 
New York, Brooklyn and Philadelphia, 
and from the Rehabilitation Clinic of 
the Jewish Hospital of Brooklyn, the Mt. 
Sinai Hospital Veterans Clinic, and the 
Post Graduate Hospital Veterans Clinic. 
All subjects were out-patients and vet- 

*Only 10 of 233 subjects who answered all 
“No’s” to this questionnaire, which was the 
normality criterion, scored over 14 on the Cornell 


Selectee Index, Form N, and only 34 of the 233 
scored over 10. 
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erans of World War II. 

Originally, only patients diagnosed by 
a psychiatrist who had treated them for 
at least several months were accepted in 
this group. However, it became impossi- 
ble to obtain a sufficient number of sub- 
jects fulfilling this criterion. As the re- 
search progressed, there was a great de- 
crease in the number of veterans diag- 
nosed as anxiety reactions since, accord- 
ing to several psychiatrists consulted, 
those patients whose maladjustment 
arose primarily out of situational diffi- 
culties in the armed service or in becom- 
ing readjusted to civilian life either re- 
covered or “bound” their anxiety in a 
specific neurotic or psychotic reaction 
pattern. Many patients who retained 
their free floating anxiety would be 
diagnosed by a psychiatrist, after a few 
months of treatment, in terms of the 
dynamic factors underlying the anxiety 
rather than as anxiety reactions. 

Finally, for those patients still diag- 
nosed as anxiety reactions, interrogation 
of the psychiatrists making these diag- 
noses generally revealed that this cate- 
gory was being used as a “waste basket” 
and that many of the patients had either 
a mixed psychoneurosis, or were possible 
latent schizophrenics. 

Therefore, in order to secure the de- 
sired number of subjects, it became 
necessary to accept a much less rigid cri- 
terion; namely, the diagnosis of a screen- 
ing psychiatrist who had interviewed 
the patient for about 45 minutes. It is 
recognized that many of the patients 
diagnosed by a screening psychiatrist 
as anxiety reactions might, after more in- 
tensive diagnostic work, be classified 
differently, However, none of these sub- 
jects may be considered normal in the 
sense of being well adjusted, relatively 
few could have been schizophrenic, and 
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all had at least a fair amount of free 
floating anxiety. 


3. Schizophrenics 


About half the subjects in this group 
came from the Veterans Hospital at 
Northport, while most of the others, 
some of whom were also veterans of 
World War II, came from Brooklyn 
State Hospital. A small number were 
being treated as out-patients in the New 
York and Brooklyn Mental Hygiene 
Services of the VA. 

For the most part, the subjects were in 
relatively good condition. Not only were 
the best patients selected from the two 
hospitals, but the patients had to be will- 
ing and able to fill out a fairly compli- 
cated test satisfactorily. (‘“Satisfactory” 
standards are described in the footnote 
in Section IV,B,2.) No coercion was used 
to make the subjects take the test, and 
even less aid was given them than under 
many group testing situations, since no 
proctor assistance was made available 
during the test. 

Although question concerning the orig- 
inal diagnosis of schizophrenia existed 
in less than a half dozen subjects, many 
of the patients were considered at the 
time of testing to be at or near remission. 
Almost half the Northport subjects were 
open ward patients, and many were re- 
leased or sent out on trial visits less than 
two weeks after taking the test. Thus, 
subjects in this category represented the 
“best” schizophrenic population; that is, 
the type of subject who often might not 
be identified in a brief interview or from 
responses to a questionnaire.* 

* A survey of the 110 Northport subjects, made 
from one to two years after testing, revealed that 
33 had been discharged, 17 were out on trial 
visit’ status, 2 were AWOL (absent without 
leave) , 5 had been transferred to other hospitals, 


and 53 remained. Thus, about half no longer 
required hospitalization. Of those remaining, 


No attempt was made to use the schizo- 
phrenic sub-classifications (catatonic, 
hebephrenic, simple, and paranoid) be- 
cause nearly all psychiatrists felt that the 
sub-classifications they gave to many of 
the subjects were unreliable. 


4. Special Characteristics of the Four Sets 
of Subjects 


a. Set Cr. While each set of subjects 
was supposed to include 50 persons in 
each of the three diagnostic categories, 
there were actually 100 normals and 53 
anxiety reactions and 50 schizophrenics 
in this set because some of the subjects 
did not complete the entire battery. How- 
ever, the combined results of the subjects 
in each category was equivalent to results 
which would have been obtained had 
there been 50 subjects in each category 
completing the test battery. 


(1). Normals. These subjects were available 
for but half an hour. Therefore, they were 
screened with only the six-item questionnaire, 
and 50 were given Tests A and B while another 
50 were given Test C. 

(2). Anxiety Reactions. Most of these sub- 
jects were thoroughly evaluated by a psychiatrist. 
There were 53 subjects, 41 of whom took all 
three tests. One subject took only Test A, two 
took Tests A and B, two took Tests A and C, 
and seven took Tests B and C, These discrep- 
ancies occurred because some subjects did not 
answer all the tests satisfactorily, while others 
came in late and missed part or all of Test A. 
Results were available for 50 subjects on Tests 
B and C, but for only 46 subjects on Test A. 
Therefore, the frequencies in the intervals in the 
distributions of the last group were multiplied 
by 50/46 to obtain comparable distributions. 

(3). Schizophrenics. All 50 subjects took 
the complete test battery. Forty-two of them were 
patients at the Northport Veterans Hospital 
while the other eight were out-patients. 


b. Set C2. There were 50 subjects in 
each diagnostic category in this set and 
in each of the two following sets. All 


many have been in good enough condition to be 
sent out on trial visit or leave of absence since 
their testing. 
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subjects completed the entire test bat- 
tery. 


(1). Normals. These subjects were first 
screened by the six-item questionnaire, and were 
given the Cornell Selectee Index, Form N, after 
completing the SMCR. Only subjects with a 
score of 10 or below on the Cornell Index were 
included in this set. 

(2). Anxiety Reactions. There were 29 sub- 
jects selected on the basis of a screening diag- 
nosis, The other 21 subjects were evaluated by 
the psychiatrist treating them. However, in many 
cases, the psychiatrist had not seen the patient 
long enough to formulate more than a tentative 
diagnostic impression. 

(3). Schizophrenics. There were 25 subjects 
from Northport Hospital, and 25 from Brooklyn 
State Hospital. 


c. Set X. 


(1). Normals. These subjects were selected 
as in Set Ce. 

(2). Anxiety Reactions. There were 41 sub- 
jects selected on the basis of a screening diag- 
nosis, while the other g subjects were given an 
evaluation, in some cases a tentative one, by 
the psychiatrist treating them. 

(3). Schizophrenics. This group included 25 
subjects from Northport, and 25 from Brook- 
lyn State Hospital. 


d. Set C3. This set was composed of 
subjects who were in excess after the 
quota for Set X had been filled, plus 
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subjects who had not quite met the 
requirements for the previous sets. 


(1). Normals. There were 10 subjects who 
met the requirements outlined in C2; g subjects 
who scored 10 or below on the Cornell Index 
but who had answered one “yes” on the six- 
item questionnaire; 28 subjects who had scored 
between 11 and 14 inclusive on the Cornell, 8 
of whom answered one “yes” and one of whom 
answered two “yes’s” to the six-item question- 
naire; and three subjects who completed only 
half of the Cornell Index but whose score on this 
half indicated that they would probably have 
scored under 10 had they completed the test. 

(2). Anxiety Reactions. There were 33 “ex- 
cess” subjects, and 17 subjects who had a sup- 
plementary diagnosis, such as “and depression” 
or “in an inadequate personality” which indi- 
cated that they were not relatively pure anxiety 
reactions. There were 18 in the former group and 
11 in the latter whose diagnosis was made by 
a screening psychiatrist. 

(3). Schizophrenics. This group included 
g out-patients, 18 open ward patients from 
Northport Hospital and 23 patients in the 
process of insulin shock treatment at Brooklyn 
State Hospital. 


Sets Ci, C2 and Cg were used as cri- 
terion populations upon whom scoring 
weights were established, while Set X 
served as the cross validation popula- 
tion. 


N ORDER to demonstrate that psycho- 
I pathology can be detected by a multi- 
ple choice projective test, it would be 
necessary to discriminate successfully 
only between two of the three diagnostic 
groups tested. Therefore, in the treat- 
ment of the data, each pair of two diag- 
nostic categories was considered separ- 
ately. Thus, a comparison was made be- 
tween the normals and the anxiety 
reactions; another comparison was made 
between the normals and the schizo- 
phrenics; and a third comparison was 
made between the anxiety reactions and 
the schizophrenics. For convenient refer- 
ence, each set of two diagnostic groups 
was termed a “comparison pair,” result- 
ing in: 

1. A normal-anxiety reaction compari- 
son pair (N-AR) 

2. A normal-schizophrenic comparison 
pair (N-S) 

3. An anxiety reaction-schizophrenic 
comparison pair (AR-S) 


Each of these comparison pairs was 
treated as a distinct unit, and the pro- 
cedure outlined below for the N-AR 
pair was duplicated for the other two. 


B. STATISTICAL PROCEDURE 


1. Method of Determining Scoring 
Weights 


The scoring weight given each res- 
ponse was determined by the relative 
percentage of subjects in each diagnostic 
group selecting that response. The pro- 
cedure followed is outlined below. 

a. The percentage of normals and anxiety 
reactions making a response was computed. 
Possible responses included the various alterna- 


lives listed in a question plus the failure to 
select any alternative. 


IV. TREATMENT OF THE DATA 


A. Dtacnostic Groups COMPARED 


b. The significance of the difference between 
these percentages was then determined according 
to the formulae outlined in paragraph c. be- 
low. 

(1). If the normals’ percentage was signifi- 
cantly higher, the response was given a weight 
of minus 1. 

(2). If the anxiety reactions’ percentage was 
significantly higher, the response was given a 
weight of plus 1. 

(3). If the difference was not significant, the 
response was weighted o. 

(In the N-S comparison pair, a significantly 
higher normal percentage received a minus 1 
weight, while in the AR-S comparison pair, a 
significantly higher anxiety reaction percentage 
received a minus 1 weight.) 

c. Two different formulae were used to de- 
termine the significance of the difference between 
the percentages. 

(1). The first was a formula adapted from 
one outlined by Guilford’ (14, pp. 298-301): 


P,—P; 


where W = Weight. When the weight came to 
plus 0.5 or above, a weight of plus 1 was given. 
Similarly, a weight of minus 0.5 or below was 
weighted minus 1. 
P,= the percentage of subjects in group 1 
making a response. 
P,= the percentage of subjects in group 2 
making a response. 
P,+ P, 
2 
q='!—p 
(2). The second was the customary formula 
for finding the significance of a difference be- 
tween percentages, namely: 


P,—P, 


*In Guilford’s formula, a constant, plus 4, is 
added so that possible weights will all be posi- 
tive and range from o through 8. Guilford’s 
formula was modified because the writer found 
it more convenient to work with plus and minus 
weights. Weights were restricted to plus or minus 
1 as the very few responses which would have 
received higher weights with the Guilford 
formula would have caused no appreciable dif- 
ference in the results. 
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where C.R. = the critical ratio, and the other 
symbols have their conventional meaning. 
With this formula, the determination of 
whether a difference is significant will depend 
upon the size of the critical ratio required. The 
higher the critical ratio required, the greater the 
difference between the two percentages will have 
to be. Four different critical ratios were used 
as criteria to determine weights as follows: 


(a) A C.R. of 1.6449, which is equivalent 
to a p of .10. 

(b) A C.R. of 1.9600, equivalent to a p 
of .05. 

(c) A C.R. of 2.3263, equivalent to a p 
of .o2. 

(d) A C.R. of 2.5758, equivalent to a p 
of .o1. 

d. Thus, there were five different weighting 

“systems” used: 

(1). That based on the formula of Guilford 
(referred to in future as weighting 
system G). 

(2). The four based on the regular per- 
centage-difference formula where the p 
required to indicate a significant differ- 
ence varied as follows: 

(a) p=.1o (referred to as weighting 
system .10). 

(b) p=.o5 (referred to as weighting 
system .05). 

(c) p= 02 (referred to as weighting 
system .02). 

(d) p=.o1 (referred to as weighting 
system .01). 


2. The Responses Weighted 


Since failure to respond to a stimulus 
is frequently a psychopathological sign 
indicative of “stimulus shock”’ or evasion 
on the individual Rorschach, such fail- 
ures were treated as regular responses. 
Thus, in Tests A and B where the sub- 
jects were requested to select one of four 
percepts, there were five possible res- 
ponses; four of them were the different 
percepts listed, while the fifth was the 
failure to select a percept. Similarly, in 
Test C, the five steps in the rating scale 
gave rise to five possible responses, while 
failure to select any rating provided a 
sixth possible response. The relatively 
few cases in which the responses were 
“spoiled” by the checking of more than 
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one percept or rating were scored as 
“failure to make a selection.” 

To make sure that failure to make a 
selection did not represent a general 
negativism toward the test situation, the 
results of the subjects who had selected 
too few of the alternatives listed were 
eliminated.? 


3. Criterion Populations 


Scoring weights were determined from 
several criterion populations as follows: 


a. Set Cr. Weights were established upon this 
population for the purpose of scoring the same 
population. It was recognized that such treat- 
ment would lead to a spuriously high differenti- 
ation since all “chance” differences were 
weighted in favor of greater differentiation. 
However, if little differentiation resulted from 
this treatment, then cross validation would be 
unnecessary, as no better results could be ex- 
pected. 

b. Sets Cz and C2 were combined to form 
population C12, This was the criterion popula- 
tion upon which weights were developed for 
cross validation on Set X. 


c. Sets Cz, C2 and C3 were combined forming 
population C123. This was also used as a cri- 
terion population upon which weights were de- 
veloped for cross validation on Set X. Population 
C123 was used to obtain information regarding 
the effect of: 


(1). Increasing the size of the criterion 
population. 


(2). Increasing the size of the critical ratio 
required for a response to be given a scoring 
weight. There were too few responses given 
weights in criterion population C12 when a p o/ 
oz or .o1 was used as a standard to permit an 
adequate judgment concerning these standards. 


2 Except for a few anxiety reactions in Set Ci, 
subjects were required to select specific alterna- 
tives in at least 60% of the questions in each 
test, and in 75% of the test battery as a whole. 
In the total population of 653 subjects, only 6 
subjects responded properly to less than 80°, of 
the questions in the battery, — 13 responded 
properly to less than 85%, and only 29 responded 
properly to less than go%. Furthermore, when- 
ever it was felt that a subject had made his 
selections capriciously, as was suspected when 
the response checks formed an arithmetic © 
geometric pattern, no data from this subjec 
were tabulated or scored. 
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4. Weighting Systems Used with Different 
Criterion Populations 
a. With population C1, only weighting system 
G was used. 


b. With population Ciz, the weighting systems 
used were: 


(1). G 
(2). p= 10 
(3). P 


c. With population C123, the weighting sys- 
tems used were: 


(1). G 

(2). p= .10 
(3). P= 05 
(4). p= 02 
(5). p= 


5. Cross Validation 


All cross validation was done on popu- 
lation X. The statistics used to evaluate 
the differences found between two dis- 
tributions were: 


a. The percentage of subjects identified in- 
correctly at the best critical score, Normal sub- 
jects scoring above the critical score were con- 
sidered incorrectly identified, as were anxiety 
reactions with scores below it. The critical score 
was placed at that point which would result in 
the minimum percentage of subjects incorrectly 
identified in the total population; that is, the 
minimum number obtained when the number 
of anxiety reactions incorrectly identified was 
added to the number of normals incorrectly 
identified. 
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Two “best” critical scores were found empiri- 
cally, one based on the distributions of the 
criterion population, and the other based on the 
distributions of the cross validation population. 
The percentage of subjects incorrectly identified 
in the cross validation population at each of 
these critical scores was determined. It is recog- 
nized that the exact percentage of subjects in- 
correctly identified at a critical score will not 
be highly reliable. However, an indication of 
the possible variability can be obtained by not- 
ing the percentage of subjects incorrectly identi- 
fied at a point a few scores higher and a few 
scores lower than the critical score. 

Furthermore, the most desirable critical score 
would vary greatly with the purpose for which 
a screening measure was used. As Ellis and 
Conrad (10) have pointed out, in most practical 
situations the percentage of false negatives or 
the percentage of false positives, as the case may 
be, is of greater importance than a combined 
percentage of wrong identifications. Thus, a 
much higher critical score would be employed 
when the purpose of the test was to definitely 
eliminate anxiety reactions than when the test 
was used to select possible anxiety reactions for 
a more thorough evaluation. However, the pres- 
ent study is attempting to determine generally 
whether any psychopathological screening is pos- 
sible with a multiple choice projective test, and 
consequently it is concerned with the largest 
amount of differentiation possible, rather than 
with the direction of the error. 


b. The significance of the difference between 
the two distributions. This was determined as a 
critical ratio. 


j 


V. RESULTS 


A. PRELIMINARY ANALYSIS UPON POPULATION Ci 


FTER population Ci had been tested, 
A a preliminary analysis of the data 
was made to determine whether further 
testing under the same experimental con- 
ditions was indicated. Scoring weights 
were developed from the responses made 
by the subjects in Ci using the Guilford 
formula (weighting system G). Then 
these weights were used to score the sub- 
jects in C1. 

Since half of the “normal” subjects in 
Ci took only Test AB, while the other 
half took only Test C, it was not pos- 
sible to obtain a full battery score for 
these subjects. Therefore, the results of 
Test AB and Test C were treated sepa- 
rately. 

A comparison of the Test AB scores 
received by the normals and the anxiety 
reactions revealed that only 9% were 
incorrectly identified at the best critical 
score. For Test C, only 6% were incor- 
rectly identified. In the normal-schizo- 
phrenic comparisons, only 8% were in- 
correctly identified with each of the two 
tests. An anxiety reaction-schizophrenic 
comparison was made only for Test AB 
where 12.5% of the subjects were incor- 
rectly identified. In all comparisons, the 
critical ratios were over 10. These results 
appeared definitely to warrant further 
testing with the SMCR. 

An analysis was made of the results 
obtained with Tests A, B and C sepa- 
rately, and with various sections within a 
test, to determine whether any modifica- 
tions should be made in the battery be- 
fore further testing was undertaken. 
However, the differences in test condi- 
tions which were investigated, such as in 
the time of stimulus exposure, appeared 
to produce little difference in the results 


obtained so that no changes were made 
in the test battery. 


B. Cross VALIDATION OF THE SMCR 


In the data presented below, unless 
otherwise specified, the results will al- 
ways pertain to the subjects in popula- 
tion X; that is, the cross validation popu- 
lation. 


1. Discrimination Obtained with Dif- 
ferent Weighting Systems 


a. Percentage of subjects incorrectly 
identified. Table 2 presents the scores 
obtained by normal and anxiety reac- 
tions subjects when different sets of 
weights are used. 

In this table, as elsewhere, N repre- 
sents the normal subjects and AR the 
anxiety reactions. Under “Score” are pre: 
sented the lower scores in each step in- 
terval. To the right of each score are the 
number of subjects found in that step 
interval. At the bottom of the page, “M” 
“SD,” and “CR” have their conventional 
meanings. The “% Wrong” stands for 
the percentage of subjects incorrectly 
identified at the best critical score. 

Tables 3 and 4, presenting the results 
of the normal-schizophrenic and anxiety 
reaction-schizophrenic comparison pairs, 
are similar in form to Table 2. 

The amount of discrimination ob- 
tained for the three comparison pairs with 
different sets of weights is summarized 
in Tables 5A and 5B which present the 
percentage of subjects incorrectly iden- 
tified (% Wrong) and the critical ratios 
respectively. 

As can be most easily observed from 
the summary tables, 5A and 5B, the 
various sets of weights produce a remark- 
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RESULTS OBTAINED WHEN THE NORMAL AND ANXIETY REACTION CROSS-VALIDATION 
POPULATIONS ARE SCORED WITH DIFFERENT SETS OF WEIGHTS 


Criterion 


Weights Based on C 12 


Weights Based on C 123 
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RESULTS OBTAINED WHEN THE NORMAL AND SCHIZOPHRENIC CROSS-VALIDATION 
POPULATIONS ARE SCORED WITH DIFFERENT SETS OF WEIGHTS 


* These calculations were made from data grouped with a class interval of 2. Because such a table was too large for one 
printed page, the class intervals were enlarged to 4 in the above table. 

+ These calculations were made from ungrouped data because in this statistic, unlike the others, grouping errors will oc- 
cur in only one direction; that is, to increase the % Wrong. 
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_* These calculations were made from data grouped with a class interval of 3. Because such a table was too large for one 
printed page. the class intervals were enlarged to 6 in the above table. 
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TABLE 4 


RESULTs OBTAINED WHEN THE ANXIETY REACTION AND SCHIZOPHRENIC CRross-VALIDATION 
POPULATIONS ARE SCORED WITH DIFFERENT SETS OF WEIGHTS 


Criterion Weights Based on C 12 Weights Based on C 123 
Weighting System Weighting System 


«10 .05 
Ss AR Ss 


n 


DWE 
HW 


HH 


HUNON COU 
> ON OWN 


-9 . ‘ 6.4 
4-7 8 8 5.2 


24 26 


* These calculations were made from data grouped with a class interval of 2. Because such a table was too large for one 
printed page, the class intervals were enlarged to 4 in the above table. 


t These calculations were made from ungrouped data because in this statistic, unlike the others, grouping errors will occur 
in only one direction, that is, to increase the oN Wrong. 


TABLE sA 


SUMMARY OF PERCENTAGE OF SUBJECTS INCORRECTLY IDENTIFIED (% WRONG) 
IN THE Cross-VALIDATION SAMPLE 


Weights Based on 
Comp. Pair Population C 12 Weights Based on Population C 123 


.05 G 10 | 02 .O1 


18 18 21 
27 27 24 


19 22 
25 31 


3! 31 3° | 3° 3! 


25 25 25 25 28 


TABLE 5B 
CRITICAL RATIOS FOR STATISTICS IN TABLE 5A 


Population C 12 Population C 123 


Comp. Pair 


16 
\ Score G .10 105 G -02 
i j AR | | AR AR Ss AR Ss 
I 
8 I 
i + 3 I 5 
° 6 14 4 17 
4 13 | 15 15 
14 Ir 17 10 
ol I 4| 13 3 
—13 8 4 
| | 
N* -3| —8.1|—2.9| 
CR* 5-3 5.1 
% Wrongt 24 | a 25 31 
| C123 
N-AR 28 29 29 31 
N-S 20 22 20 20 
AR-S 24 24 25 27 
Mean | 25 | | 25 | 26 
a Mean | Mean 
| G | .10 | .05 | G | .10 | .05 .02 .O1 | Cra | C123 
N-AR 4-3 4-5 4.1 3-4 3-9 3-4 2.7 3.8 4-3 3-4 
N-S 9.3 6.9 7.9 7.2 7.1 7.0 6.8 6.1 7.3 6.8 
AR-S 5.3 4:7 4-3 4.8 4.8 5.2 5.1 g.3 4.8 5.0 
Mean 5.6 | 5-4 | 5-4 | 5-3 | 5.2 4-9 5.0 | §.1 


ably similar amount of discrimination. 
For each comparison pair, the middle 
six (of eight) “% Wrong” figures have a 
range of only three percentage points, 
while the critical ratio range is less than 
one point. This consistency clearly indi- 
cates that differences both in the weight- 
ing systems and in the criterion popula- 
tions upon which the weights are based 
have had little effect upon the amount of 
discrimination produced. 

b. Particular subjects incorrectly iden- 
tified. It is possible that while the same 
percentage of subjects was incorrectly 
identified by each set of weights, there 
may be differences with respect to the 
particular subjects who were incorrectly 
identified. Therefore, a comparison of 
the subjects incorrectly identified by each 
set of weights was made. 

For each diagnostic group in a com- 
parison pair, there were eight distribu- 
tions, a different one produced by each 
set of weights. The 10 subjects scoring 
lowest or highest, whichever indicated 
incorrect identification, were selected 
from each distribution as the ones most 
clearly misidentified. Then, for each 
diagnostic group in a comparison pair, 
the 10 subjects appearing in most distri- 
butions as incorrectly identified were 
considered the criterion “incorrectly 
identified groups.” It was found that 
there was very little difference between 
the 10 subjects selected by any of the sets 
of weights and the subjects in the criter- 
ion group. 

Because of the marked similarity in 
the results obtained with the various 
sets of weights, it was felt justifiable to 
use only one set in the subsequent treat- 
ment of the data. The weighting system 

= .10 with weights derived from popu- 
lation Ci2g was selected because it iden- 
tified so many of the subjects in the 
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incorrectly identified group and because 
it led to the widest range in scores, an 
important factor when the results ob- 
tained from only sections of the test bat- 
tery were considered. 


2. Effect of Increasing the Number 
of Subjects 


The fact that adding the subjects in 
population Cg to the criterion popula- 
tion did not increase the validity may be 
attributed to one of two reasons. Either 
the 100 subjects in population Cig pro- 
vided a sufficiently large number to at- 
tain the best possible set of weights, or 


the poorer diagnostic classification of the 


subjects in Cg was enough to balance 
any additional validity to be gained from 
adding 50 subjects to each category in 
the criterion group. To investigate this 
problem, populations Ce and C3 were 
scored separately with weighting system 
p = .10 based on criterion population 
Ci23. Population Ci was not scored be- 
cause a total score could not be obtained 
for the “normal” subjects since each of 
them had taken only half of the testing 
battery. 

In the N-AR comparison pair, the 
percentage of subjects incorrectly identi- 
fied was 18% for Ce and 16% for Cg. In 
the N-S comparison pair, these percent- 
ages were 16% and 17% respectively, 
while for the AR-S comparison pair 
they were 19% and 25% respectively. 
While it is true that Ce and Cg each 
formed one-third of the criterion popula- 
tion upon which the weights were based, 
still if the diagnostic classification of the 
subjects in Cg was materially inferior, 
this population should have shown much 
less discrimination. Since the percentage 
of subjects incorrectly identified was very 
similar for both Ce and Cg, it would 
seem more logical to conclude that for 
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this test the diagnostic classification of 
subjects in Cg could not be much in- 
ferior to that of the subjects in C2. 
Hence, for the SMCR, the addition of 
subjects to the criterion population be- 
yond 100 per diagnostic category would 
not seem likely to materially increase 
the amount of discrimination produced. 


3. Reliability of the SMCR 


Because subjects were not available for 
retesting, the reliability was determined 
by a measure of internal consistency. 
The test battery was divided into four 
sections: Test A, Test B, the first half of 
Test C, and the second half of Test C. 
Each section was then split into two 
halves according to a table of random 
numbers. Scores on one half of each 
section were summed to give a Part 1 
score which was correlated against a 
Part 2 score derived from the sum of 
scores on the other half of each section. 

For each of the three comparison pairs, 
reliabilities were found for each diagnos- 
tic group separately, and then for both 
groups combined. In the N-AR com- 
parison, the reliability coefficients, when 
corrected by the Spearman-Brown proph- 
ecy formula, were .69 + .o7 for the nor- 
mals alone, .75 +.06 for the anxiety 
reactions alone, and .77 + .o4 when both 
groups were combined. In the N-S com- 
parison, the corrected coefficients were 
.78 + .05 (N), .82 +.05 (S), and 87 + 
.o2 (N-S); while those for the AR-S 
comparison were .65 + .o8 (AR), .67 + 
.o8 (S), and .72+.05 (AR-S). These 
results would indicate that the SMCR 
has a reasonable degree of reliability. 


| 4. Validity of the SMCR. 


There would seem to be no question 
that valid group differentiations have 
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been made for each comparison pair. 
In Table 5B only one of the critical 
ratios was under 3.0, and that came to 
2.7. However, the amount of error when 
individual prediction is desired can be 
gathered only from the percentage of 
subjects incorrectly identified. In Table 
5A, when weighting systems p=.o1 was 
eliminated because it was the only one 
which gave somewhat atypical results, 
the median of the other seven weighting 
sets indicate that 30% of the subjects in 
the N-AR comparison pair are incor- 
rectly identified, as are 20% in the N-S 
comparison pair and 25% in the AR-S 
comparison pair. 

However, it must be recognized that 
these percentages are based on the “best” 
critical score of the cross validation dis- 
tributions. This procedure weights all 
sampling errors in favor of a greater 
discrimination, and thus gives a spuri- 
ously high validity. To properly estimate 
validity, the critical score should be 
based on a previous population. There- 
fore, the best critical score was found 
from the distributions of population C23 
scored with weighting system p=.10 
based on criterion population C1ig3. 
These critical scores proved to be +2 for 
the N-AR comparison pair, —21 for 
the N-S comparison pair, and either 
—7 or —8 for the AR-S comparison 
pair. There were two possible critical 
scores for the last pair because both gave 
the same percentage of subjects incor- 
rectly identified, 

At the above cutting scores there 
would be 30% of the N-AR and 25% 
of the N-S comparison pairs incorrectly 
identified. Since a cutting score of —8 
would result in 31% of the subjects being 
incorrectly identified, while a score of 
—7 would result in 29% of the subjects 
being incorrectly identified, it would ap- 


pear best to estimate the percentage of 
subjects incorrectly identified in the 
AR-S comparison pair at 30%. 

It should be recognized that using the 
results of only one rather than a series of 
samples to establish the critical scores, 
and having too few subjects in the distri- 
butions for smooth curves may lead to 
an underestimate of the validity. There- 
fore, the best possible estimate of the 
validity may lie somewhere between the 
above results and those obtained when 
using the best critical score of the cross 
validation distribution. However, exactly 
where in between these points it would 
lie is impossible to determine, and it 
would appear safest to conclude that the 
SMCR will incorrectly identify at most 
30% of the N-AR comparison pair, 
25% of the N-S comparison pair, and 
30% of the AR-S comparison pair. 


5. Classifying Subjects According to 
Tri-Scores 


In the above estimate of validity, the 
subjects in a comparison pair were scored 
only with the set of weights derived from 
the distributions of that comparison 
pair. However, these subjects could also 
be scored with the weights derived from 
the other two comparison pairs. Thus, 
it was possible to score each subject with 
three sets of weights, one from each com- 
parison pair, and thereby classify him: 

1. As normal (N) or as anxiety reac- 
tion (AR) from the results obtained with 
the N-AR comparison pair scoring key. 

2. As N or S from the N—S compari- 
son pair scoring key. 

3. As AR or S from the AR-S com- 
parison pair scoring key. 

Each subject could then be classified 
in one of the following eight ways: 
(These classification combinations will 
be referred to in the future as tri-scores.) 
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1. As N on the N-AR key; N on the N-S 
key; and AR on the AR-S key. NNA 
2. N (N-AR key); N (N-S key); S (AR-S 


key). NNS 
3. N (N-AR_ key); S (N-S_ key); AR 
(AR-S key) . NSA 
4. N (N-AR key); S (N-S key); S (AR-S 
key). NSS 
5. AR (N-AR key); N (N-S key); AR 
(AR-S key). ANA 
6. AR (N-AR key); N (N-S key); S$ 
(AR-S key). ANS 
7. AR (N-AR key); S (N-S key); AR 
(AR-S key). ASA 
8. AR (N-AR key); S (N-S key); S$ 
(AR-S key). ASS 


To determine the validity obtained by 
using these tri-scores, the same validation 
procedure was followed as when subjects 
in a comparison pair were scored only 
with the set of weights derived from that 
comparison pair. 

Population Cz was used as the criter- 
ion population, and the tri-scores of this 
group were found. The diagnostic iden- 
tification of these subjects was based on 
whether they scored above or below a 
critical score. For each comparison pair 
the critical score was determined from 
the distributions of the two diagnostic 
groups within that comparison pair. (In 
all comparison pairs, the point at which 
the critical score fell proved to be the 
same for population Cg as had been 
found for population C23.) 

For the cross validation population, 
two separate tri-score distributions were 
obtained; one where the diagnostic clas- 
sification of the subjects was based on a 
critical score derived from the distribu- 
tion of the cross validation population 
itself (Pop. X), and the other where the 
diagnostic classification was based on the 
point selected as the critical score in 
population Ce. (Pop. X’), The tri-score 
distributions are presented in Table 6. 

The tri-score weights were determined 
for each comparison pair by the tri-score 
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TABLE 6 


NUMBER OF SuBJECTS IN TrI-SCORE CLASSIFICATION 


Population C2 Population X | Population X’ 
(Classified according to (Classified according to (Classified according to 
Tri-Score | own “best” critical score) | own “‘best”’ critical score) C2 critical score) 

NNA 14 3 32 19 8 
NNS oo" I ° I I ° r ° I 
NSA “aap I 2 4 5 4 6 | I 6 
NSS | 6 ° 14 6 ° 20 I ° 12 
ANA I 8 ° 2 6 ° 4 9 3 
ANS ° ° wd ° I ° ° I ° 
ASA ° 20 3 | Ss tog 3 3 12 7 
ASS 3 8 | 20 3 8 13 


distributions of the two diagnostic 
groups in a comparison pair. For ex- 
ample, in the N-AR comparison pair, 
the tri-score NNA was considered as a 
normal score since the population Ce 
section of Table 8 reveals that 39 nor- 
mals and only 12 anxiety reactions made 
this tri-score. For two tri-scores, NSA and 
ANS, there were the same number of 
normals and anxiety reactions making 
these scores. In such cases, the tri-score 
was weighted according to the results of 
the comparison pair involving the two 
groups concerned. Thus, in the N-AR 
comparison, the results obtained with 
the N-AR set of weights would deter- 
mine whether a subject would be classi- 
fied as normal or as anxiety reaction. 

It can be seen in Table 7 that for both 
group X and X’, the % Wrong using 


TABLE 7 


only the critical score in a subject’s com- 
parison pair gives about the same results 
as the % Wrong using the tri-score. The 
very limited improvement which results 
when the best possible combination of 
tri-scores is found for each group indi- 
cates that little could be expected from 
using a criterion population larger than 
Ce in establishing the tri-score weights. 

However, the tri-score is not entirely 
without advantage. Thus, for subjects 
receiving the tri-score NNA, a statement 
could be made that they are most likely 
to be normals; but that if they are not 
normals, then they are more likely to be 
anxiety reactions rather than schizo- 
phrenics. There are a sufficient number 
of subjects with tri-sscores NNS, ANA, 
ASA, and ASS to make similar interpre- 
tations possible for subjects in these clas- 


PERCENTAGE OF WRONG DISCRIMINATIONS OBTAINED WITH CRITICAL SCORES 


ONLY, AND WITH TRI-SCORES 


Population C2 


Population X 


Population X’ 


N AR 


N AR Ss 


With critical score only _ 18 16 
At best tri-score combination 17 15 
At tri-score combination using 


C2 tri-score wts. 


19 30 21 24 30 25 31 
17 20 20 22 30 25 27 


30 | 24 31 25 29 


| 


sifications. Of the 134 subjects in popu- 
lation X’ receiving these tri-scores, a pre- 
diction as to which of the three diagnostic 
groups a person is not in would be incor- 
rect in only 17, or 13%, of the cases. Had 
the diagnostic prediction of the 16 sub- 
jects not receiving any of the above tri- 
scores been made on a chance basis, the 
percentage incorrectly identified would 
still be under 17%. 


6. Comparison of A Priori Judgments 
as to Diagnostic Significance of Res- 
ponses with Empirical Results 


The a priori judgments were com- 
pared to the empirical results in regard 
to the diagnostic implications (normal, 
anxiety reaction or schizophrenic) of the 
various percepts. The results of this com- 
parison are presented in Table 8. Listed 
at the left are the diagnostic categoriza- 
tions which were determined empirically. 
These categorizations were based on the 
results obtained with the entire experi- 
mental population. The letter “O”’ indi- 
cates those percepts which did not re- 
ceive any weights (using the p= .10 
weighting system), and hence could not 
be considered characteristic of either 
diagnostic group. 

The a priori judgments as to diagnos- 
tic implication are presented along the 
top of the table. The “?” stands for the 
alternative “something else,” while the 
letter “F” is the response “failure to se- 
lect one of the alternatives in a question.” 
Separate results were computed for Test 
AB and Test C. In the former, it was 
possible to determine the empirical 
weight for each percept exactly. In the 
latter, however, this determination had 
to be made qualitatively. 

In Test C, the subjects checked one 
of five possible ratings for each percept, 
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and a separate weight was found for 
each of these five ratings. For some per- 
cepts, the diagnostic implication was 
quite clear since the first few rating steps 
were characteristic of one diagnostic 
category, while the last few were charac- 
teristic of the other diagnostic group. 
However, the picture generally was more 
confused. For example, the first and 
fifth steps might be given a normal 
weight, the fourth step an anxiety reac- 
tion weight, and the second and third 
steps given no weight. With results such 
as these, it was impossible to define the 
percept as characteristic of normals or of 
anxiety reactions. In many cases, border- 
line decisions had to be made, such as 
when of the five rating steps only the 
first or second step was found to be 
weighted. Included in the present table 
were only those percepts whose diagnos- 
tic implications seemed fairly clear. 

Table 8 reveals that for the N-AR 
comparison pair there is no correlation 
between a priori judgment and the em- 
pirical results.? 

Of the 60 percepts considered to be 
characteristic of normals on Test AB, 
only g proved to be so empirically, and 
in four cases the supposed normal per- 
cepts were found to be characteristic of 
anxiety reactions. Of the 80 percepts 
judged to be characteristic of anxiety 
reactions, only 6 were so empirically, 
while 13 were found to be characteristic 
of normals. Thus, in only 15 of the 140 
percepts did the a priori judgment coin- 
cide with the empirical evaluation, while 
in 17 cases they were exactly the op- 

*It should be clearly recognized that the au- 
thorities who estimated the psychopathological 
significance of the responses did so on the basis 
of their experience with the individual Ror- 
schach, and they made no claims that the sig- 


nificance would remain the same on a multip 
choice test. 
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posite. Very similar results were also ob- 
tained on Test C. 

For the other two comparison pairs, 
there appears to be a positive correlation 
between a priori judgment and empiri- 
cal evaluation. In the N-S comparison, 
of the 60 percepts presumed to be nor- 
mal on Test AB, 16 actually were so with 
only 5 being exactly the opposite; that 
is, being characteristic of schizophrenics. 
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TABLE 8 


29 percepts coincide, and 25, are exactly 
the opposite. 

It is difficult to use a single statistic to 
indicate the amount of correspondence 
because a lack of coincidence between 
the a priori judgment and the empirical 
results may be expressed in two ways: 
the estimated percept may prove to be 
significant of neither diagnostic category; 
or, what appears to be more serious, it 


CoRRESPONDENCE BETWEEN EMPIRICAL AND A PriorI CATEGORIZATIONS 


A Priori Categorization 


Comparison 
Pair 


Empirical 
Categorization 


Test AB 


N 


F 


Normal 9 13 
N-AR Anxiety Reaction 4 6 
Neither 47 61 


6 2 ° 14 12 2 ° 
19 4 I 4 5 16 5 
55 14 59 


Normal 16 19 
N-S Schizophrenic 5 12 
Neither 39 49 


6 4 ° 17 8 6 ° 
29 5 35 13 9 14 67 
45 II 25 


Anxiety Reaction 14 15 
Schizophrenic 3 15 
Neither 43 50 


10 2 ° 18 19 I ° 
14 I 26 12 14 23 45 
56 17 34 


Of the 80 percepts estimated as schizo- 
phrenic, 29 were found to be so empiri- 
cally with only 6 having exactly the op- 
posite diagnostic significance. Thus, in 
45 of the 140 percepts the a priori judg- 
ments and the empirical results coin- 
cided, while they were exactly the oppo- 
site for only 11 percepts. The results of 
Test C, though, are not quite as good, 
with only 31 of the percepts coinciding 
while 19 were exactly the opposite. 

In the AR-S comparison it is Test C 
which shows the greater amount of cor- 
respondence between a priori judgment 
and empirical evaluation. Here 42 of the 
percepts coincide, with only 15 being 
exactly the opposite. However, little 
relationship is shown in Test AB where 


may turn out to be significant of exactly 
the opposite diagnostic category than 
was expected. 

In general, it may be said that a posi- 
tive correlation exists between a priori 
judgments and empirical results since 
these two coincide more frequently, on 
the whole, than they are exactly the op- 
posite. However, this correlation seems 
to be fairly low as can be seen from the 
relatively few times a percept is found 
empirically to have exactly the same 
significance as was expected on an a 
priori basis. Thus, while a priori judg- 
ments may be of value in the initial se- 
lection of items for a multiple choice 
projective test, there can be no question 
but that it will be essential that the 


psychopathological significance of each 
percept be determined empirically. 
However, the data in Table 8 also in- 
dicate that a detailed analysis of those 
percepts found to be significant of a diag- 
nostic category may uncover general 
principles applying to multiple choice 
projective tests which could materially 
increase the validity of a priori judg- 
ments. For example, in the N-AR com- 
parison, the a priori estimations would 
have been much more accurate had those 
percepts which were considered to be 
characteristic of schizophrenics been 
judged to be indicative of anxiety reac- 
tions, while those percepts predicted as 
typical of anxiety reactions were instead 
considered characteristic of normals. 
This holds true for both Test AB and 
Test C. Similarly, in the AR-S com- 
parison, the percepts which were 
thought to be indicative of anxiety reac- 
tions proved to be much less discrimi- 
nating than the percepts which has been 
assumed to be characteristic of normals. 
Also apparent from Table 8 is the fact 
that the alternative “Something else” 
(?) does not appear to differentiate any 
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TABLE 9 
CoMPARISON OF CONDITIONS VARIED WITHIN THE TEST BATTERY 
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diagnostic group, while the “Failure to 
select an alternative” (F) seems to be 
primarily a schizophrenit response. 


7. Comparison of Conditions Varied 
within the Test Battery 


As outlined previously (Section II,B, 
2), the experimental design included 
variations in certain of the test condi- 
tions. By comparing the results of differ- 
ent sections of the test battery, the effect 
of these variations could be determined. 
The relative discrimination obtained 
under the different test conditions is 
shown in Table 9. While the amount of 
discrimination under each condition was 
measured by both the critical ratio and 
the % Wrong (percentage of subjects in- 
correctly identified), only the critical 
ratios are listed because of the extremely 
high correlation between these two sta- 
tistics. Preference was given to the criti- 
cal ratio because it would seem to be the 
more reliable statistic. 

a. Differences in the structure of the 
questions. 

(1). Relative vs. absolute decision 
by the subject. In Test A and Test B, 


Critical Ratios for Different 
Condition Test Section Comparison Pairs 
N-AR N-S 
Relative vs. absolute decision | Test AB (relative) 3-4 5-5 4.0 
by subject Test C (absolute) 3.0 7.8 4.1 
Difference in type of percepts | Design I (forced normal) 0.7 4.1 2.7 
in a question Design II (unlimited) 3.8 3-7 2.8 
Design III (forced non-normal) 1.6 4.6 3.0 
Briet ~s. lengthy exposure of | Test A (brief) 2.6 4.1 3-5 
stimulus Test B (lengthy) 3.2 5-9 2.5 
No. of questions in the test | Designs I, II, II1 (20 questions) 2.0 4-5 2.8 
battery Test A, B (30 questions) 2.9 5.0 3.0 
Test AB (60 questions) 3-4 cx 4.0 
Na C (100 questions) 3.0 7.8 4.1 
est ABC (160 questions) 3-9 7.2 4.8 
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which have been combined to form Test 
AB, the subjects were required to select 
one of four alternatives, thus making a 
relative decision among these alterna- 
tives. On Test C, the subjects rated each 
alternative on a five point absolute rat- 
ing scale. As can be seen from the results 
in Table 9, the type of decision made by 
the subjects appears to have little effect 
upon the amount of discrimination. 
While Test AB is somewhat superior for 
the N-AR comparison pair, Test C has 
a higher critical ratio for the N-S com- 
parison pair, while the results are prac- 
tically identical in the AR-S compari- 
son pair. 

(2). Differences in the types of alter- 
natives included in a question. Results 
were obtained for three different ques- 
tion “Designs,” each of which was struc- 
tured to include only certain types of 
percepts (section II,B,2a [2]). It was 
necessary to select percepts on the basis of 
an a priori judgment as to their psycho- 
pathological significance, and, as has 
been shown, such a priori judgments do 
not correspond to the psychopathologi- 
cal significance found empirically. 
Hence, the original rationale for this 
comparison no longer exists. However, 
there remains a comparison between 
questions in which the subjects are re- 
quired to select a specific percept (Designs 
I and III), and questions in which the 
subject may avoid selecting a specific 
percept by checking the alternative, 
“Something else,” (Design 

From the results it is seen that Design 
II was superior for the N-AR compari- 
son pair, while Design III was best for 
the N-S comparison pair. Little differ- 
ence was found among the designs in 
the AR-S comparison pair. While no 
superiority seems to be shown for either 
the “forced-choice’’ or the “unlimited- 
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choice” question designs, any conclu- 
sions should be made. cautiously since 
these results were based on only 20 ques- 
tions of each design. 

b. Differences between a brief and 
lengthy exposure of the stimulus. In 
Test A the stimuli were exposed for a 
relatively brief period. In Test B, which 
was equivalent to Test A in other re- 
spects, the subjects were given as much 
time as they desired to view the stimulus. 
As shown in Table 9, Test B produced 
greater discrimination in the N-AR 
and N-S comparison pairs, while Test 
A was superior in the AR-S compari- 
son pair. These results do not appear to 
be definitive enough to warrant drawing 
any general conclusions. It remains pos- 
sible that the differences found represent 
true differences, and that opposite results 
are obtained in different comparison 
pairs because “optimum” conditions are 
different for each pair of diagnostic 
groups. 

c. The effect of tncreasing the length 
of the test. By considering different sec- 
tions of the test battery separately, it was 
possible to determine empirically the 
effect of increasing the length of the test. 
Designs I, II and III each included only 
20 questions, so the mean of the critical 
ratios obtained with these “Designs” was 
used to estimate the discrimination pro- 
duced by 20 questions. The amount of 
discrimination resulting from go ques- 
tions was determined by the mean of the 
critical ratios obtained with Test A and 
Test B, each of which contained that 
number of questions. When Tests A and 
B were combined to form Test AB, there 
was a total of 60 questions. In Test C 
there were 100 questions, while the com- 
plete battery, Test ABC, contained 160 
questions. The fact that the types of 
questions were different for different test 
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crimination. 


lengths was not considered contaminat- 
ing since it would appear from the pre- 
ceding results that the type of question 
has little effect upon the amount of dis- 


As can be seen in Table g, there is, in 
general, a steady increase in discrimina- 
tion for each comparison pair as the 
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Wrong of the three comparison pairs 
drops from 35% for the 20 question sec- 


tion to 25% for the 160 question battery. 


TABLE 10 


COMPARISON OF SUBJECTS INCORRECTLY IDENTIFIED AND SuByECTS CORRECTLY IDENTIFIED, 
WITH RESPECT TO AGE AND AMOUNT OF EDUCATION 


8. The Effect of Age and Education 


Since there were some differences 
among the three diagnostic groups with 


respect to age and amount of education, 


Age Comparison 


Education Comparison 


Diagnostic | Compar. Gr 
oup 
Petit Le N | Mn | Mdn | N* | Mn | Mdn 
N-AR Incorrect 10 25.2 25.2 10 11.7 12.5 
Correct 40 25.8 24.3 40 12.5 12.7 
Normals N-S Incorrect 10 25.5 23.5 10 11.6 II.7 
Correct 40 25.7 25.1 40 12.5 12.8 
Both Pop. X 50 25.6 24.7 50 12.3 12.7 
Total Pop. 250 25.0 23.8 245 13.2 12.8 
Incorrect 10 33-3 33.0 10 12.3 13.0 
Correct 40 30.1 29.5 40 10.6 10.6 
Anxiety AR-S Incorrect 10 39.1 36.0 10 8.8 8.8 
Reactions Correct 40 29.6 29.5 40 11.4 12.1 
Both Pop. X 50 | 30.7 30.5 50 10.9 10.9 
Total Pop. 203 | 30.0 29.6 197 11.5 
N-S Incorrect 10 27.3 25.5 8 12.3 12.0 
Correct am | one 28.5 34 II.0 11.6 
Schizophrenics | AR-S Incorrect 10 | 26.6 25.5 9 11.8 11.5 
Correct 40 | 20.4 28.8 33 11.1 11.6 
Both Pop. X 50 | 28.8 28.0 42 11.3 11.6 
Total Pop. 200 | 28.4 27.3 179 11.6 


* For some subjects information as to the amount of education was not available. 


length of the test increases. Were the 
critical ratios of the three comparison 
pairs to be averaged, then the few 
fluctuations would be eliminated, and 
the amount of discrimination would 
show a regular progression. The impor- 
tance of the length of the test upon indi- 
vidual prediction can be gathered from 
a comparison of the percentage of sub- 
jects incorrectly identified. The mean % 


it was possible that the discrimination 
obtained between diagnostic groups was 
a reflection, in part at least, of these age 
and educational differences. In Table 
10, the correctly and incorrectly identi- 
fied subjects (defined in section V,B,1,b) 
in each comparison pair are differenti- 
ated with respect to age and amount of 
education. For purposes of comparison, 
data are also presented for all of popula- 
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tion X, which includes both correctly 
and incorrectly identified subjects, and 
for the entire experimental population. 

Each diagnostic group is included in 
two comparison pairs. The results of the 
normals and schizophrenics are very 
similar in both comparison pairs because 
in each case many of the same subjects 
were incorrectly identified in both com- 
parison pairs. Six of the normals who 
were incorrectly identified in the 
N-AR comparison pair were also in- 
correctly identified in the N-S compari- 
son pair. With the schizophrenics, there 
were nine subjects duplicated. However, 
none of the anxiety reactions was incor- 
rectly identified in both comparison 
pairs. (A chance matching of 10 names in 
each diagnostic category would have re- 
sulted in two subjects being included in 
both comparison pairs.) 

There seems to be no difference in age 


between correctly and incorrectly iden- 
tified normals. However, the incorrectly 
identified schizophrenics are a few years 
younger than those correctly identified, 
while the incorrectly identified anxiety 
reactions, particularly in the AR-S com- 
parison pair, are an older group. 


The incorrectly identified normals 
have slightly less education, while the in- 
correctly identified schizophrenics are 
slightly better educated. With the anx- 
iety reactions, however, the educational 
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differences are marked. Those incorrectly 

identified as normals have about two 
more years of education than those cor- 
rectly identified, while the anxiety reac- 
tions identified as schizophrenics have 
about three years less education. 

From these results it would appear 
that, particularly in the case of the anxi- 
ety reactions, the subjects’ responses are 
a product of their educational back- 
ground as well as of their psychopatholog- 
ical condition, The educational back- 
ground may, in turn, be a reflection of 
intellectual endowment. The entire ques- 
tion appears to be too complex to be 
resolved in the present study, particu- 
larly since the inadequacy of the original 
diagnostic classification may also be a 
factor. The chief screening psychiatrist, 
in charge of the panel which diagnosed 
almost all the anxiety reactions in the 
cross validation population, stated that 
most diagnostic errors probably occurred 
when attempting to determine whether 
the older patients with little education 
(i.e., those anxiety reactions identified on 
the SMCR as schizophrenics) were anxi- 
ety reactions or schizophrenics. In any 
case, the overlap between correctly and 
incorrectly identified subjects with re- 
spect to age and education was so great 
that differences in these factors cannot 
be responsible for most of the discrimina- 
tion obtained with the SMCR. 


PON cross validation, the SMCR in- 
U correctly identified at most 30% of 
the N-AR comparison pair, 25% of the 
N-S comparison pair, and 30% of the 
AR-S comparison pair. For all com- 
parison pairs the critical ratios were 
sufficiently high to establish a significant 
difference between diagnostic groups. By 
using tri-scores, the diagnostic category 
a subject was not in could be predicted 
with an error of under 17%. Therefore, 
it would appear that a fair validity had 
been obtained with a multiple choice 
projective test. Since the amount of dis- 
crimination found compares favorably 
with the validity reported for other 
screening measures, and since there are 
indications that subsequent research 
with multiple choice projective tests will 
lead to even greater discrimination, 
further investigation with this approach 
would seem warranted. 


A. COMPARISON WITH OTHER SCREENING 
DEVICES 

As mentioned previously, screening 
tests have been found to be most effective 
in military screening. Ellis and Conrad 
(10) have presented a detailed summary 
of the results obtained with personality 
inventories in this area. For most stud- 
ies, the amount of validity reported is 
approximately the same as that obtained 
with the SMCR. While there were a few 
reports of unusually high validity, Ellis 
and Conrad suggest that such validity 
coefficients reflect the operation of un- 
usual experimental or statistical factors. 

The possible effect of improper cross 
validation, an experimental error found 
in many studies, has been well illustrated 
by data obtained in the present investi- 
gation. When a population of 50 sub- 
jects in each diagnostic group (C1) was 
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scored with weights derived from the 
same population, the scores from just 
half the test battery resulted in only 
about 8% of the subjects being incor- 
rectly identified. When weights derived 
from a criterion population of 150 sub- 
jects in each diagnostic category (C123) 
were used to score one-third of these sub- 
jects (C2 and Cg), the percentage of 
subjects incorrectly identified rose to an 
average of 18.5%. When the weights 
derived from this criterion population 
were applied to a cross validation group, 
the percentage of subjects incorrectly 
identified at the best critical score of the 
cross validation group became 25%. 
When a critical score derived from the 
criterion population was employed, the 
average percentage of subjects incorrectly 
identified rose to 28%. 

Zubin (60), in another article evaluat- 
ing screening tests, presents the results 
obtained at the Sheepshead Bay U. S. 
Maritime Training Station as an ex- 
ample of the marked effectiveness of a 
questionnaire in military screening 
which generally has produced better re- 
sults than civilian screening. Of 1,000 
men processed, 660 were passed without 
a further interview on the basis of a 
screening personality questionnaire. 
Three of those passed had to be dis- 
charged from the service during their 
training period for psychiatric reasons. 
Of the 340 men selected for a psychiat- 
ric evaluation, 15 had to be discharged 
from the service, 75 had temporary or 
mild emotional conditions which re- 
quired short term therapy, while the re- 
maining 250 turned out to be false posi- 
tives. 

In evaluating the above data, the 75 subjects 


who adjusted after short term therapy are diffi- 
cult to place. There may have been an equal 
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proportion of subjects among the persons passed 
without a psychiatric interview who had com- 
parably mild emotional conditions, but who 
were able to adjust without any short term 
therapy. If these 75 subjects were therefore to 
be considered as false positives, there would be 
a total of 1,000 subjects, g82 passing and 18 
being discharged. Of these, 33°% (325 passing 
and 3 discharged) were incorrectly identified. 
Eliminating the 75 subjects from the data would 
lower the percentage of subjects incorrectly 
identified to 27%. 

Not only are the above percentages about the 
same as those obtained with the SMCR, but 
they probably give a spuriously high estimate 
of validity. Only persons making high question- 
naire scores would have been discharged (on 
the basis of a psychiatric examination) without 
participating in the regular training program. 
Ellis and Conrad report a study by Weinstock 
and Watson “on 121 Naval recruits who were 
allowed to remain in service (‘on trial’) despite 
an adverse prognosis based on clinical judg- 
ment. Of the 121 recruits, only 44 (36%) were 
discharged for neuropsychiatric reasons during 
recruit training” (10, p. 386). 

Another factor which frequently leads 
to spuriously high validity estimates is 
the use of extreme groups. With such 
populations almost any screening meas- 
ure can obtain a fair validity. As Huston 
and Shakow found in a study on schizo- 
phrenics, while the patient groups per- 
formed at a level significantly poorer 
than the control group on two indices 
of steadiness, “when these patients who 
cooperated as well as the normal group 
were compared with the latter, the differ- 
ences between the groups disappeared” 
(25, p. 120). It was for this reason that in 
the present study care was taken to test 
only the “best” schizophrenics, and to 
use anxiety reactions who were not so 
severely disturbed as to warrant hospital- 
ization, 

Had a typical institutionalized popula- 
tion been used, the SMCR would un- 
doubtedly have produced a much greater 
amount of discrimination. For example, 
there were 48 schizophrenic subjects who 
filled out Tests A and B correctly ac- 
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cording to the established criteria, but 
who failed to fill out Test C properly. 
The greater failure on Test C resulted 
from the fact that it required a some- 
what more complex response and came 
last in the testing battery when increased 
boredom and fatigue had set in. When 
the scores of the 48 schizophrenics who 
failed to complete Test C were substi- 
tuted for the regular schizophrenic cross 
validation population in the Test AB 
comparisons, the percentage of subjects 
incorrectly identified at the best critical 
score dropped from 24% to 10% in the 
N-S comparison, and from 32% to 22% 
in the AR-S comparison. Furthermore, 
had the SMCR been administered so 
that a higher proportion of schizophren- 
ics would have responded, such as under 
individual conditions or with proctoral 
assistance, the percentage of subjects in- 
correctly identified would undoubtedly 
have been even smaller. 


B. INDICATIONS FOR GREATER POTEN- 
TIAL DISCRIMINATION 


While there has been a tremendous 
amount of experimentation with ques- 
tionnaires extending back to the first 
World War, relatively little work has 
been done with multiple choice projec- 
tive tests. Indications exist from the pres- 
ent study that further research with the 
multiple choice projective approach may 
lead to a marked increase in the amount 
of discrimination produced. 

Firstly, the present study did not at- 
tempt to exhaustively sift questions until 
a test composed of the most discrimina- 
tory choices was evolved. By far the 
greater number of responses received no 
weights, even under the most liberal 
weighting system. Yet the greater amount 
of discrimination occurring as the size 
of the test was increased clearly indicates 
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that the replacement of non-discrimina- 
tory percepts should add even more to 
the validity. 

Secondly, only one projective method, 
the Rorschach, was used. It is recognized 
in individual psychopathological testing 
that much more validity can be obtained 
from a battery of different projective 
tests than from any single instrument. 
As Rapaport and Schafer have stated: 
“.. . aS an exclusive basis of diagnosis 
the test (the Rorschach) has serious limi- 
tations and its real potency is exploited 
only when used as the center of a battery 
of tests” (42, p. 73). 

Thirdly, a loss in validity undoubtedly 
occurred because of inadequate diagno- 
ses of subjects in the three diagnostic 
classifications. The Cornell Index, which 
was used as the criterion of normality is 
itself only a rough screening device, and 
it has not proved too effective in the 
screening of prepsychotic and_ early 
psychotic states. Norms for Form N1 of 
the Cornell Index could not be obtained. 
However, 10 or fewer poor responses on 
Form Ne which contains 37 additional 
questions and consequently provides 
more opportunity for poor responses, 
will be made by 14% of the persons re- 
jected for military service for psychiatric 
reasons. Thus, it is probable that some 
subjects in the “normal” population 
were poorly adjusted. Furthermore, some 
of the schizophrenics were in remission, 
or at least well enough adjusted to be 
released from the hospital within a few 
weeks of their testing, while others had 
a great deal of free floating anxiety char- 
acteristic of anxiety reactions. The anxi- 
ety reaction group was perhaps the most 
poorly diagnosed, containing many indi- 
viduals who would not have contem- 
plated psychiatric treatment had they 
not been veterans, as well as persons 


whose anxiety was symptomatic of a 
latent schizophrenia. Almost all the 
anxiety reactions in the cross validation 
population were selected only on the 
basis of a screening or tentative psychia- 
tric diagnosis. 

Of course, most subjects in each diag- 
nostic category were probably correctly 
classified. However, the number of sub- 
jects incorrectly identified by the test was 
relatively small, so that the inclusion of 
some misclassified persons in this group 
would make an appreciable difference. 
It is recognized that the misclassified 
subjects may not be the persons incor- 
rectly identified by the test. However, 
the possibility appears to be sufficiently 
great to warrant a more thorough evalua- 
tion of the subjects in each diagnostic 
category before the maximum potential- 
ity of the test can be estimated.* 


*Since a year had ela between the testing 
of the subjects and the preparation of this 
monograph for publication, an attempt was 
made to check the accuracy of the diagnosis 
established at that time by a follow-up study. 
No further information could be obtained for 
the normal subjects since they had filled out the 
tests anonymously. However, psychiatric progress 
reports were available for most of the anxiety 
reactions and schizophrenics. 

The majority of the anxiety reactions tested 
were subsequently treated by psychiatrists work- 
ing in private clinics under contract to the Vet- 
erans Administration. These psychiatrists pre- 
pared monthly reports summarizing the condi- 
tion of their patients. The majority of the schizo- 
phrenics had been tested at the Veterans Hos- 
pital, Northport, L. L, and for these subjects 
complete clinical case folders were available. 

Unfortunately, a careful inspection of this 
data was not helpful in improving the validity 
of the original diagnosis. While almost all of 
the subjects retained their initial diagnosis, a 
consideration of their symptomatology revealed 
much overlapping between subjects diagnosed 
as anxiety reactions and those diagnosed as 
schizophrenics. It appeared that psychiatrists at 
out-patient clinics generally preferred to make 
a diagnosis of anxiety reaction while those at 
neuropsychiatric hospitals were more likely to 
make a diagnosis of schizophrenia. This is in 
agreement with the experience of the writer at 
both types of installations, and with the opinions 
of several psychiatrists consulted on this ques- 
tion. Thus, subjects with identical symptoma- 
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Finally, a multiple choice projective 
test combined with other types of tests 
in a battery may result in a greater 
amount of discrimination than would 
exist for any of the tests separately. For 
example, a test such as the SMCR which 
resulted in greater discrimination be- 
tween normals and psychotics might be 
combined with a questionnaire such as 
the Cornell Index which is more effective 
in differentiating normals from neu- 
rotics. Subjects might also be given an 
intelligence test, and their multiple 
choice projective test responses inter- 
preted according to their I.Q. It is recog- 
nized that responses on the free associa- 
tion Rorschach will depend upon a sub- 
ject’s intelligence as well as his psycho- 
pathological condition. A “vague” or 
“mediocre” response which might be 
considered normal for a subject of low 
intelligence would be indicative of mal- 
adjustment in one of high intelligence. 
The same would appear to be true of the 
multiple choice Rorschach. In the pres- 
ent investigation, the large difference in 
education between the anxiety reactions 
identified incorrectly as normals and 
those identified incorrectly as schizo- 
phrenics probably reflects a difference in 
intelligence rather than in education per 
se. Therefore, it might be advisable to 
develop different sets of scoring weights 
for different levels of intelligence. 


C. SUGGESTIONS AS TO FUTURE RESEARCH 


From the results of the present study, 
both quantitative and qualitative, a 


tology might be diagnosed differently at different 
installations. 

A further attempt to validate the SMCR on 
schizophrenics by comparing the scores of sub- 
jects who had been discharged with those who 
still remained in the hospital was abandoned 
when it was found that in a population of the 
“better” schizophrenics, discharge was deter- 
mined as much by the patient’s home environ- 
ment as by his psychiatric condition. 


number o! suzvestions arise as to the 
course of future investigations. 


1. Subjects 


In the present study, weights based on the 
results of only 50 subjects in each diagnostic 
category proved inadequate. On the other hand, 
a criterion population of 150 subjects per diag- 
nostic category was no more effective than one 
of 100 subjects. But caution would be necessary 
in applying these results to other investigations. 
For example, a smaller number of subjects might 
be required when only two or three responses 
are possible for each question rather than the 
5 or 6 in the SMCR, or when a psychopathologi- 
cal variable less heterogeneous than diagnostic 
categories is investigated. However, the lack of 
very marked differences between diagnostic 
groups in their responses to any question in 
the SMCR indicates that a fairly large number 
of subjects will probably be required to establish 
reliable scoring weights. 

Not only should the subjects in both criterion 
and cross validation populations be very com- 
petently diagnosed, but it would be valuable 
to have them evaluated psychiatrically on a 
wide variety of variables. In addition to noso- 
logical classification, the subjects could be rated 
upon significant personality traits, such as hos- 
tility or dependence, and upon dynamic factors 
such as sibling rivalry or parental rejection. 
Then, the same subject population could be 
regrouped accordingly when studying each dif- 
ferent psychopathological variable. A prognostic 
index could also be developed by reconsidering 
the data after the progress of the subjects had 
been observed, 


2. Stimuli 


Further use of the Rorschach inkblots as 
stimuli in preference to other inkb!ots would 
seem inadvisable. As the present study has 
shown, there is little correlation with respect to 
the psychopathological significance of responses 
between a priori judgments based on the free 
association Rorschach and the results of the 
multiple choice Rorschach. Therefore, the large 
experiential background which has been buill 
up with these particular inkblots under free 
association test conditions would be of little 
value in the construction of a multiple choice 
test. As a disadvantage, the multiple choice 
Rorschach provides suggestions to the subjects 
by listing possible responses which may influence 
their performance on a subsequent free associa- 
tion Rorschach.2 


2 For this reason, the scoring stencils for the 
SMCR have not been included in the present 
report. It was felt that the availability of these 
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While no quantitative evidence is available, 
it was the experience of the writer and of sev- 
eral other clinical psychologists who tested sub- 
jects individually subsequent to the SMCR that 
the performance of these subjects had been 
influenced by the preceding multiple choice 
Rorschach, The evaluation of the subjects’ free 
association protocols was hindered by the diffi- 
culty in assessing the direction and extent of the 
influence of the multiple choice test. 

Another advantage in using other than the 
Rorschach inkblots is the opportunity to con- 
struct inkblots particularly suited for a multiple 
choice test. Such blots could be structured to 
emphasize responses to either large areas or the 
blot as a whole. In the present experiment, the 
subjects rarely studied the slides minutely, but 
based their responses on a broad general im- 
pression, (Of course, it is possible that a more 
detailed examination of the stimuli would have 
been made had the subjects been more highly 
motivated or had the inkblots been presented 
in booklet form.) The inkblots could be struc- 
tured to emphasize certain determinants, such 
as Levy’s graduated “movement” series (61), or 
they might present a choice between determi- 
nants such as by containing two possible per- 
cepts, one form determined and the other 
predominantly color determined. It may be 
desirable to draw inkblot designs with printer’s 
ink, as was done by Terman and Miles (52), 
rather than to use actual inblots. While it would 
be desirable to explore the potentialities of in- 
definite figures placed on simple backgrounds, it 
is very possible that better results are obtained 
when both figure and ground are indefinite, as 
with inkblots. This reservation should be kept 
in mind for all types of projective stimuli. 

In addition to inkblots, various other pro- 
jective stimuli should be investigated. Here 
again, it would not appear advisable to use the 
same stimuli as in the free association tests, 
but rather to develop new stimuli which to- 
gether with the multiple choice responses could 
be structured in the direction of a specific 
psychopathological variable, For example, rather 
than employing the TAT cards, special pictures 
could be drawn.? To investigate the maternal 
relationship, a picture of a mother and a child, 
the former with her arm upraised, might be 
shown. The responses could include one in 
which the mother is petting the child, another 
in which she is about to strike the child, and 


stimuli might lead to an indiscriminate use of 
the test, thus exposing many sons to sug- 
gested responses with the Rorschach blots. The 
scoring stencils will, of course, be made available 
to any qualified psychologist who wishes to per- 
form a confirmatory or further exploratory ex- 
periment. 
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a third in which she is waving to someone be- 
hind the child. Of course, the picture would 
have to be sufficiently vague so that the subjects’ 
responses would depend upon their projections. 
With the Rosenzweig Picture Frustration Test 
(43, 44) which is already structured to measure 
a specific variable, that of frustration tolerance, 
it might be necessary only to develop a set of 
multiple choice responses for each of the pic- 
tures. 

Geometric figures, such as those employed in 
the Bender Gestalt (1) and the Franck Symbolic 
Sex Preferences (11) could be used. Here, the 
alternatives might consist of other geometric 
figures, with the subjects required to select the 
figure most closely resembling the stimulus. 
Perhaps particularly well suited for the detec- 
tion of paranoid tendencies would be the Tauto- 
phone (13) in which a recording of low-voiced 
nonsense syllables is played. Subjects are asked 
what the recording is saying, being given to 
understand that it is producing a meaningful 
statement. 

Further work can be done with story plots 
along the lines already developed by Kelly, and 
Bishop, Taylor and Wolfenstein (27, 50, 59). 
Verbal stimuli could also be confined to single 
words in a variation of the word association 
approach. Either fragmented letters or very brief 
exposures during which only one or two letters 
would stand out might be employed, with sub- 
jects being asked which word had been dis- 
played. 


3. Administration of the Test 


It would seem desirable to present the stimuli 
as tests of ability, and in fact, to include some 
questions with obviously correct responses to 
further this belief. The best exposure time would 
have to be worked out separately for each dif- 
ferent stimulus. While the results of the present 
experiment indicated that the length of exposure 
of the Rorschach slides made little difference 
in the amount of discrimination obtained, no 
conclusion can be reached as to whether this 
will hold true for other stimuli or for other 
psychopathological variables which may be in- 
vestigated. 

It may be possible to diminish the effect of 
suggestion and rationalization in the selection 
of responses and thereby obtain more significant 
projections by removing the stimulus before the 
multiple choices are presented to the subjects. 
For example, in the SMCR this could be done 
by listing the percepts in a question on a second 
slide which would be projected on the screen 
after the stimulus had been removed. The sub- 
jects could indicate their choice of response by 
marking the appropriate column on an IBM 
answer form, With this technique, the subjects 
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might be forced to reply more upon their initial 
free associative responses to the stimuli. 


4. Structure of the Questions 

No definitive conclusions could be reached 
from the results of the present study regarding 
the structure of the questions. Similar results 
were obtained when the subjects made a relative 
decision by selecting one of four percepts and 
when they made an absolute decision by rating 
a percept on an absolute five point rating scale. 
Differences due to variations in question “de- 
sign” (that is, the types of percepts contained 
in a question) could not be investigated as 
originally planned because the psychopathologi- 
cal significance of the percepts had been deter- 
mined on an a priori basis, and there proved to 
be little correlation between a priori judgment 
and the empirical results. However, little dif- 
ference was found between “forced choice” 
questions in which subjects had to select a 
specific percept, and “unlimited choice” ques- 
tions in which they could check the alternative 
“Something else.” 

Because of the lack of relationship between a 
priori judgment and empirical findings, it would 
seem advisable in subsequent experimentation 
to determine first, insofar as is possible, the 
psychopathological significance of each response 
empirically from a simple three or four point 
rating scale. Then, after the significance has 
been established empirically, the percepts could 
be fitted into the structures of different question 
designs. These designs could extend from the 
relatively simple one suggested by Malamud and 
Malamud (32) in which the subjects select either 
one of two percepts listed or the alternative 
“neither,” to a design with as many as 10 
percepts, as in the Harrower Multiple Choice 
Rorschach (19). 
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5. Length of Test 


It would seem desirable to have the test as 
long as possible. The present experiment demon. 
strated that the amount of discrimination in- 
creased with the length of the test up to at least 
160 questions, and it may very well continue 
beyond that point. With the use of a variety 
of stimuli rather than a repetition of the same 
1@ inkblots, the subjects should not become 
bored or fatigued as quickly. Moreover, it was 
felt that the discriminating power of a stimulus 
decreased as additional questions were applied 
to that stimulus. Therefore, using a higher pro- 
portion of stimuli to questions than in the 
SMCR may also serve to increase the validity. 


6. Scoring of the Test 


Since the five weighting systems employed in 
the present investigation all produced about the 
same amount of discrimination, it would not 
seem to make much difference which weighting 
system were to be used. However, in addition 
to the simple weighting of items, it may be 
advisable to make a Guttman scale analysis (16) 
of the responses. Few psychopathological vari- 
ables are “pure” entities. This is particularly 
true of nosological categories, For example, psy- 
chiatrists now prefer to speak of the “schizo- 
phrenias” rather than of “schizophrenia,” be- 
lieving that this diagnosis encompasses a variety 
of only loosely associated conditions. Similarly, 
many divergent psychopathological factors may 
lead to the symptomatic diagnosis of anxiety 
reaction. A scale analysis may disclose a relatively 
small combination of responses which are patho- 
gnomonic of a particular type of psychopathology 
but which do not materially affect the subjects’ 
total score when all questions are weighted 
equally. 


I. Further exploration with the multi- 
ple choice projective test approach in the 
detection of psychopathology would seem 
warranted since: 

A. A fair validity has ben obtained 
with a multiple choice projective test. 
When scoring-weights based on a sample 
of 100 subjects in each diagnostic cate- 
gory were cross validated on a new 
sample of 50 subjects in each category, 
the percentage of subjects incorrectly 
identified by the SMCR was at most 
30% for the normal-anxiety reaction 
comparison, 25% for the normal-schizo- 
phrenic comparison, and 30% for the 
anxiety reaction-schizophrenic compari- 
son, There was an error of under 17% 
in predicting which of the three diag- 
nostic categories a subject would not be 
in, 

B. Split half reliability coefficients, 
when corrected by the Spearman-Brown 
prophecy formula, were .77 in the nor- 
mal-anxiety reaction comparison, .87 in 
the normal-schizophrenic comparison, 
and .72 in the anxiety reaction-schizo- 
phrenic comparison. 

C. There are grounds for believing 
that even greater validity may be ob- 
tained by: 

(1). Further experimentation with 
various multiple choice projection tests. 
(2). Combining a multiple choice 
projective test with a questionnaire and 


VII. CONCLUSIONS 


an intelligence test in a screening battery. 

II. Of the different test conditions 
varied in the present experiment, only 
the length of the test was shown to be 
significantly related to the amount of 
discrimination obtained. 

III. No significant differences in the 
amount of discrimination could be 
established from: 

A. The application of different 
methods of weighting responses. 

B. Increasing the size of the cri- 
terion population upon which the 
weights were developed from 100 to 150 
subjects in each diagnostic category. 

C. The subjects responding by mak- 
ing a “relative” as opposed to an “ab- 
solute’’ decision. 

D. The subjects being forced to 
choose among specific percepts in a 
question as opposed to being given the 
“unlimited” alternative, “Something 
else.” 

E. Exposing the stimuli for a brief 
period as opposed to a lengthy period. 

IV. There was little correlation be- 
tween a priori judgments based on the 
free association Rorschach and the em- 
pirical results obtained with the SMCR 
with respect to the psychopathological 
significance of responses. 

V. Both age and education, the latter 
perhaps a reflection of intelligence, may 
affect the subjects’ responses. 
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NAME 


VIII. APPENDIX 


A. Tue SINGER MULTIPLE CHOICE RORSCHACH 


Test A 
AGE 


1 
wolf's head 
_____An emblem 
A topographical map 
—__—An octopus 


— 4 — 
_____A bomb bursting 
—__——Lungs and blood 
Two dogs 
____— Something else 


7 — 
Two red roosters 
Ribs 
—__—Lungs and chest 


_____Flag of a country 
Mountain range 
A black waterhole 
Spilled ink 


—___A pair of antlers 

—___A butterfly with torn 
wings 

—____Dried blood 

_____Something else 


— 16 — 
—__—A spattered mess 
_____A totem pole 
backbone 
———A 


_____Intestines 
formation 
squashed frog 
—___A topographical map 


bears climbing 
___—<A biology slide 
—_____A sailboat 
—____Something else 


2 — 
Ruins of a castle 
A blot 
A squashed bug 
—__—An X-ray picture 


— 5 — 
______Animal heads 
Torn up butterfly 
———A map 
Bears’ heads 


wih 

____Vertebra 

______X-ray of the hips 

Person tearing someone 
apart 

____Blood and black 


A cow with fur 
—__— Two feet 
—____An X-ray 
___‘Something else 


DATE. 


3 
An animal split open 
Line down the center 
butterfly 
else 


____..Two equal sides 
Black and red paint 
___—Dark clouds 
—__—A large eye 


9 
Smoke 
____A hair ribbon 
Nothing at all 
_____Something else 


Necks of geese 
Two scottie dogs 
A spinal column 
Sex organs 


horrible object 
_____A map 

______A butterfly 


___An oil explosion 
______A bad design 

______A cut up male sex organ 
___Black paint 


_____Two heads 
A coastline 
_____Nothing at all 
______Something else 


__.__Head of small dog 


______Two polar bears climbing 


___—An Easter egg 
—__ Modern art 
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A line down the center 

___—A black mess 

Nothing at all 


insect 
______Jack the Ripper 
_____A riverbed 
______Something else 


—_____A butterfly 
clown’s face 
People watching each 
other 


______Colored butterflies 
An X-ray picture 
airplane 
_____A blotch of colors 


nter 
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—25— — 26 — 
______Flowers —___—A dirty mess 
clouds —__——Smoke rising __——A spinal column 


_____ Birth of a child A bridge in the distance 
____ Sea horses —___Nothing at all 


______ Something else 


— 28 — — 30— 
_____Male sex organs and blue and yellow blue feathery flower 
Leaves Bones A terrible germ 
_.____Bugs Two spiders ____Spiders attacking 
_____People hanging in air —___Something else somebody 


—_._——Deer running 


Test B 


person cut up 


A halloween mask 


NAME AGE GRADE DATE 

—1iA— —2A — 3A 
_—A Oat Burnt piece of paper ____—A coastline 
______Female sex organs ———A spine __——A dark cloud 
staring mask cat's face 

____Something else —__—Two people —— 

—4A— —5A— —6A— 
—— ee ___—Human chest with a ____ Entrance to a tunnel 
__.__Two people hole in it _____Nothing at all 
___—<A pair of pliers A squashed bug _____A white lamp 
_____Smoke —_____Blood 


______Something else 
A coastline 


—8A— —9A— 
_____Sex organs —___Waiters bowing ____Two devils 
____A skeleton effect ______A symmetrical design _____._Two monkeys hanging 
__....Two staring eyes ______Animal cut open down 
Picture from a medical -_____Something else spinal cord 


book 


Birds’ heads 


—10\ — —12A— 

giant walking towards <A dragon skin relief map 
you ———A man sitting down « Clouds 
— Two snakes —_—A gorilla torn chest 
A furry man —______Snakes 

—_ Something else 

— 14A — —15A — 
— of a mouse buffaloes crushing ____Dark smoke 
—__Profile of a man a rabbit — 
______Two equal sides _____._The nervous system 

Islands 


— 16A — 


—17A— — 18A — 
—____A cat ___-Nothing at all Head of a dog 
—__—An X-ray Picture from an airplane _ A dragon fly 
—___A squashed worm __—Flying geese A cloud 
—_____Water ______Something else ______A furry snake 
—19A — — 20A — —21A— 
Pacific island rabbit’s head Smoke 
__.Two women go siping __——Clouds ___.Just an inkblot 
__.Male sex organs ______Scottie dogs large eye 
— ~ Something else —_—Small dots on an island 


______Fog or mist 
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—22A— — 2gA — — 24A — 
spinal column ___..Pink, blue, orange ____Two fish 
______Decorated Christmas tree Snow covered mountain in a medical 
______Flowers ______.Male sex organs book 
Ribs ____A mountain _ ___A coat of arms 
_. Something else 
—25A— — 26A — —27A — 
famous painting liver ___Horns of a deer 
_____Staring eyes _____Blood ____.An explosion 
__.__Red, green, orange ___Two women ____Intestines 


Drawing in a medical ___Something else man’s face 
book 


— 28A — — 29A — — 30A — 
____.Squashed insects __.....( tobacco pipe A Christmas tree 
wishbone ___Two spiders —_—__Modern art 

___Snakes eating a rabbit's __.__Deer running ______Bloody insides 


head Mountain sides ______INustration of different 
—____ Something else flowers 


ONLY A NOT 
EXACTLY MUCH SOMEWHAT LITTLE AT ALL 
LIKE IT LIKE IT LIKE Ir LIKE IT LIKE I! 


. Two Santa Clauses 

. Nothing at all 

A bet .... 

. Female sex organs 

Woman with her arms raised 
. A staring mask 

. Small islands 

. A squashed bug 

. Ruins of a castle 


2 
3 
4. 
5 
6. 
7 
8 
9 


. A man with his head cracked open .____ 
. A butterfly 

. Two clowns playing pattycake 

. Nothing at all 

. Entrance to a tunnel 

. Faceson the edges ................. 

. Two equal sides 

. An arrowhead 
. A large eye 

. A white lamp in the center .......... 


— 


= 
. A person tearing someone else apart. .- 
. A bloody mess .. 
. A coastline , 
. A symmetrical design ......./......-— 
. Waiters bowing .................... a 
. Men picking something up . . 


Test C 
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VERY ONLY A NOT 
EXACTLY MUCH SOMEWHAT LITTLE AT ALL 
LIKE IT LIKE IT LIKE IT LIKE IT LIKE IT 


31. Head of an animal ................- 
34. A skin stretched out tight ............ 
36. A black waterhole .................— 


. A giant walking towards ee 


. The mesveus system .....-......-+.. te 
42. A line down the center .............- 


54. Picture from an airplane ........... 


r6. A cat’s whiskers ...................- 


. An oil explosion ...................— 
. Rug of animal skin ................- 


— VII — 
66. Two female faces ................-.- bite 


. Rocks balancing 


— — 
71. Animals climbing 
72. A coat of arms 


= 

il 
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VERY ONLY A 
EXACTLY MUCH SOMEWHAT LITTLE 
LIKE IT LIKE TT LIKE IT LIKE IT 


. An evergreen tree 
. Colored butterflies 

. Modern art 


. Nothing at all 


. A burning candle 

. Blood 

. Two men in dunce caps 
. Two women 

. Smoke rising 

. Bloody hands 

. A colored map 

. Lobsters 


. A biology slide 


. A wishbone 

. Nothing at all 

. Snakes eating a rabbit’s head 

. Flowers 

. Two lions 

. Two people hanging in air 

. Squashed insects 


B. Stx-ITEM QUESTIONNAIRE TO SCREEN NORMALS 
Answer each question YES or NO. If you are not sure, guess. 


. Have you ever had treatment for a nervous condition? 

. In general, do you consider yourself a nervous person? 

. While in the service, were you ever hospitalized for psychoneurosis, combat fatigue, nervousness 
or some similar condition? 

. Since you have gotten out of the service, have you generally (not merely occasionally) felt 
irritable or jumpy? 

. Do you get upset more easily than the average person? 

. Have you ever filed a disability claim or collected a pension for psychoneurosis? 
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